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editorial 


Examining the Leaky STEM Talent Pipeline—Need for Further Research 


A college education has long been considered the key to 
the American dream, opening the door to a vista of oppor- 
tunity and financial prosperity. Data on college access in 
recent decades indicate that the United States has made 
significant strides in providing that key to students across 
the nation. With the establishment of federal programs 
such as the Pell Grant and TRIO Programs targeted to 
assist low-income students, it is tempting to conclude that 
all students, regardless of socioeconomic background, 
face a level playing field in terms of college access and 
completion. In reality, although recent years have seen 
significant improvements in college access for U.S. stu- 
dents as a whole, gains in access continue to lag for low- 
income students, and those who do enroll in college are 
consistently less likely than their higher-income peers to 
secure baccalaureate degrees (Adelman, 2006; Engle & 
Tinto, 2008; Kena et al., 2015). A recent study conducted 
by the National Student Clearinghouse Research Center 
revealed that over the past two decades, over 31 million 
students enrolled in college without graduating with a 
degree or certificate. Indeed, for students in the low- 
income demographic, the world of higher education con- 
tinues to look less like a ladder to success and more like a 
revolving door. Along with low-income students in a 
variety of majors, this revolving door continues to displace 
students who choose science, technology, engineering, and 
math (STEM) majors, a phenomenon that has implications 
not only for individual students, but also for the growing 
income inequality and STEM labor force in our nation. 

STEM fields, according to the Bureau of Labor Statis- 
tics, fall within two major domains: (a) the science, engi- 
neering, mathematics, and information technology 
domain, including the fields of life sciences, physical sci- 
ences, and social sciences; and (b) the science- and 
engineering-related domain, including the fields of health 
and architecture (Jones, 2014). Employment in these 
STEM fields is not only projected to grow faster than the 
average for all occupations, but the median wage of STEM 
workers in 2013 was more than double that of other occu- 
pations: (Vilorio, 2014). These high-paying jobs fulfill a 
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labor force need and represent a path out of poverty for the 
low-income demographic for students with STEM bacca- 
laureate degrees. In fact, the President’s Council of Advi- 
sors on Science and Technology identified as a national 
priority bolstering the number of students earning a STEM 
undergraduate degree by 34% over 2012 rates (Olson & 
Riordan, 2012). The focus on increasing the numbers of 
STEM graduates seems warranted when one considers 
that, although between 2003 and 2009, 28% of first-year 
bachelor’s degree students declared a STEM major, nearly 
half of those students eventually left their STEM studies, 
either by switching majors or leaving college altogether 
(Chen & Soldner, 2013). This attrition was unevenly 
divided between income quartiles, however, and between 
2003 and 2009, over 29% of STEM majors in the lowest 
income quartile left college without a degree or certificate 
compared with only 15% of STEM majors in the highest 
income quartiles (Chen & Soldner, 2013). 

As the plethora of literature on college access, student 
persistence, and student success suggests, these are 
complex enterprises, spanning a range of disciplines, from 
sociology to economics to psychology to education (Perna 
& Thomas, 2006; Terenzini & Reason, 2005). Students’ 


college experiences are as varied and nuanced as the stu- 


dents themselves, and numerous contextual factors beyond 
the need for financial support contribute to students’ deci- 
sions to enroll in and complete college degrees and must 
be considered in any investigation of student success. Fur- 
thermore, although three decades of research helped us 
identify disproportionate factors contributing to student 
attrition from the postsecondary pipeline, translating 
theory into actionable models and effective programs has 
proved difficult. We still lack scalable models for student 
access to and persistence while attending college (Tinto, 
2007). 

It is clear that college access, student persistence, and 
overall student success (including graduation and place- 
ment in the workforce) are arduous challenges in the 
United States, particularly for economically disadvan- 
taged students in STEM fields. While some entities have 
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periodically evaluated the programs focused on increasing 
the talent pool and the number of STEM degree seekers 
and completers; there has not been a close examination of 
investments, strategies, and associated impact of on the 
STEM pipeline in terms of STEM student persistence, 
retention, completion, and placement into the workforce. 
Much can be learned from the significant investment that 
has made on providing access to low-income students 
from all backgrounds (i.e., ethnic/racial, disability, 
gender), including examining the academic and career tra- 
jectories (i.e., majors/programs, career placement) that 
students choose, as well as the barriers/challenges they 
encounter along their educational path. 

Carla C. Johnson 

Purdue University 


Janet B. Walton 
Purdue University 
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There is the tendency to explain away successful urban schools as indicative of the heroic efforts by a tireless 
individual, effectively blaming schools that underperform for a lack of grit and dedication. This study reports the 
development of a research instrument (School Science Infrastructure, or SSI) and then applying that tool to an 
investigation of equitable science performance by elementary schools. Our efforts to develop a science-specific instru- 
ment to explore associations between school-level variables and equitable science performance are informed by James 
Coleman’s tripartite notion of social capital: the “wealth” of organizations is encompassed within their social norms, 
informational channels, and reciprocating relationships. Grounded in school effectiveness research and social capital 
theory, the instrument that we report on here is a valid and reliable tool to support meso-level investigations of factors 


contributing to school variations in science achievement. 


The persistence of science achievement disparities is not 
an intractable problem. Even when populated by similar 
students, certain schools overcome the “demographics is 
destiny” stereotype. Unfortunately, there is the tendency to 
explain away successful urban schools as indicative of the 
heroic efforts by a tireless individual (Tough, 2008)— 
effectively blaming schools that underperform for a lack 
of grit and dedication. In contrast, empirical studies of 
school effectiveness demonstrate how math and reading 
achievement gaps are narrower in schools displaying par- 
ticular organizational and leadership structures (Bryk, 
Sebring, Allensworth, Luppescu, & Easton, 2010). From 
our vantage point, educational equity reforms, including 
those specific to science, solely focused on classroom 
interventions will falter if contextual factors are ignored 
(Ladson-Billings, 2006). Just as Petty, Wang, and 
Harbaugh (2013) reported in this journal how schoolwide 
characteristics are related to mathematics achievement, the 
current study used the school as the level of analysis with 
the goal of identifying causes of—and potential solutions 
to—science achievement gaps (National Center for Edu- 
cational Statistics [NCES, 2011]). A crucial step toward 
realizing this goal is to develop techniques for studying 
school-level aspects of science education. Here we report 
on the development of an instrument to assess infrastruc- 
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tures associated with elementary school science and dem- 
onstrate how it can be applied to address pressing 
questions about equity in science education. 


Background 
The controversy about school effects on student learn- 
ing can be traced to the legendary 1966 Coleman Report. 
In that document, family and neighborhood factors were 


‘shown to be substantially more influential on children’s 


learning than did anything transpiring within the school 
(Coleman, 1968). Those findings prompted considerable 
educational reform ranging from school desegregation via 
busing to countless studies of linkages between school 
practices and learning outcomes (Cuban, 1984; Viadero, 
2006). Bifurcated beliefs about causes for uneven student 
achievement continue to be debated. A commonly 
endorsed view is that learning is largely a psychological 
process: children acquire knowledge and skills supplied by 
their classroom teacher and the instructional materials 
(Allen et al., 2013). The associated logic is that differences 
in achievement between children can be attributed to 
student aptitude and teacher talents. Research on science 
learning by and large endorses this notion. Most science 
education research about student cognition concentrates 
on individual mental activity despite widespread 
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acceptance of social constructivist theory. Resistance to 
the growing evidence that organizational features shape 
children’s science learning belies a psychological parochi- 
alism. Piagetian views associated with Sputnik-era reform 
maintain a grip on science education researchers despite 
evidence that canonical views (e.g., developmental stage 
theory) do not withstand empirical analyses (Metz, 1995). 
A more accurate perspective about learning is that students 
jointly construct meanings even as broader background 
factors shape what youngsters are able to achieve 
(Bransford, Brown, & Cocking, 1999). Consequently, 
achievement differences can be attributed to factors that 
transcend the individual. 

Anyone who has taught science recognizes how the 
work environment, the surrounding community, and the 
interactions among people influence. efforts to support 
student learning. Research has repeatedly demonstrated 
how school organization and leadership affect student 
achievement (Hallinger & Heck, 2011; Wahlstrom & 
Louis, 2008). However, science education researchers 
seem resistant to the notion that context matters by instead 
choosing to primarily conduct research on the interplay 
between a teacher and his or her students, with little atten- 
tion paid to contextual factors that exert subtle yet sub- 
stantial influences. A hasty dismissal of organizational 
influences on students’ science learning is not only short- 
sighted but could undermine efforts to reduce science 
achievement gaps (NCES, 2011). 

None of this is meant to suggest that the quality 
of teaching is inconsequential—far from it (Nye, 
Konstantopoulos, & Hedges, 2004). Given the multiple 
culprits for inequitable education (i.e., poverty rates, 
uneven distribution of educational resources, imbalances in 
the availability of quality teachers, inadequate supplies of 
quality curriculuny,and the variability in learning environ- 
ments), it is reasonable that science educators would focus 
on areas where they are most familiar and comfortable 
(Darling-Hammond, 2010). When achievement disparities 
are acknowledged, attention is typically directed at teacher 
preparation, professional development, and curriculum 
refinements. Neglected is the sizable evidence that school 
structure and leadership practices—the ecology of a school 
and the coordination of that system—have demonstrable 
impacts on students’ academic performance (Leithwood, 
Harris, & Hopkins, 2008). Unfortunately, when school 
effectiveness research is conducted, almost invariably it is 
restricted to the heavily tested subjects of reading and math 
(Bryk et al., 2010), with few studies about the contextual 
impacts on science achievement (Fraser & Kahle, 2007; 
Wood, Lawrenz, Huffman, & Schultz, 2006). 
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If we are genuinely concerned that schools are perpetu- 
ating discriminatory practices, a claim advanced specifi- 
cally for science education (Schmidt & McKnight, 2012), 
then we must be open to an array of transgressing agents. 
Resolving achievement disparities necessarily requires 
attending to the wide range of contributing variables— 
from instructional strategies to curriculum materials to 
out-of-school influences (Evans, 2005; Milner, 2012; 
Rodriguez, 2001). One challenge is that, even with 
the availability of student science test scores, to date 
school effectiveness studies of science education have 
been hampered by the absence of science-sensitive 
instrumentation. Because school leadership and organiza- 
tional structures have content-specific features (Spillane, 
2005), additional research tools are needed. This article 
reports on the development of such an instrument that can 
then be applied to the investigation of associations 
between school-level factors and equitable science 
performance. 


School Organization and Leadership 

Although few studies examine connections between 
science learning and school effectiveness, considerable 
research has revealed strong statistical relationships 
between school contexts and student achievement in 
reading and math (Coelli & Green, 2012; Leithwood, 
Patten, & Jantzi, 2010). School organization and 
building-level leadership would be reasonably expected 
to influence science achievement just as has been discov- 
ered for math and reading. However, this line of investi- 
gation is full of challenges, given the complexities of 
distinguishing student features from broader school-level 
factors. Poverty serves as a prime example of this conun- 
drum. At a micro-level, a child living in poverty would 
be expected to have more difficulty in science classes 
than a child who lives in a nonimpoverished home. Being 
poor imposes challenges in the form of insecurities: 
uncertain supplies of nutritious food, transiency/ 
homelessness, and the associated psychological and 
developmental diminutions (Ransdell, 2012). In contrast, 
a meso-level consideration of poverty affects schoolwide 
achievement as measured by the proportion of low- 
income children enrolled (NCES, 2010). This distinction 
is equivalent to comparing cellular activity with ecologi- 
cal processes. Rather than concentrating on what tran- 
spires in one classroom, we emphasize the likelihood that 
schools as organizational systems and the leadership 
exerted within those settings produce observable and 
substantial influence on student science outcomes. 
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Schools as Organizations 

Researchers who value schoolwide influences on 
student achievement have noted the importance of parental 
involvement, an orderly atmosphere, an’ achievement- 
oriented climate, and staff. cohesiveness (Scheerens, 
2001). Van de Grift and Houtveen (2006) found moderate 
effect sizes for organizational features that distinguished 
improving schools: teacher collaboration, faculty meet- 
ings centered on educational quality, and supportive 
school climate. In addition, leadership in the form of 
administrative support and encouragement of the faculty 
correlated with significant achievement growth. In con- 
trast to beliefs about schools’ weak impacts relative to 
family and community inputs, such studies reveal how 
schools enhance students’ academic performance. 

Multiple school-level factors are consistently shown to 
be viable contributors to not only improved student 
achievement but also reduced inequities in test perfor- 
mance. Among the contributing factors are organizational 
trust (Cosner, 2009), professional capacity (Bryk et al., 
2010), and social networks (Penuel, Riel, Krause, & 
Frank, 2009)—especially the influence of leaders within 
the school (Hallinger & Heck, 2011; Wahlstrom & 
Louis, 2008). The sizable impact that schoolwide factors 
can have on equitable test performance resonates with 
recent re-analyses of the Coleman Report’s data 
(Konstantopoulos & Borman, 2011). Numerous school 
effectiveness studies confirm linkages between organiza- 
tional characteristics of a school and the achievement of 
students (Supovitz, Sirinides, & May, 2010). However, 
despite a host of proposed organizational models (e.g., 
Griffith, 2003) the field has yet to identify a theoretical 
framework that adequately accounts for these findings 
(Scheerens, 2013). 


Leadership Within a School 

Within an ecological, school-level perspective, the orga- 
nization can be thought of as the structure, while leader- 
ship practices serve to coordinate the system. Principals 
have myriad responsibilities that include school building 
operations (scheduling, space management), financial 
oversight (budgets, grants), personnel duties (supervising, 
hiring), instructional guidance (teaching evaluations, pro- 
fessional development), internal coordination (faculty 
cohesiveness), and external relationships (family partner- 
ships, corporate sponsors) (May, Huff, & Goldring, 2012). 
When the time a principal spends on various tasks is 
examined in light of improved test performance, intriguing 
patterns arise (Horng, Klasik, & Loeb, 2010). Principals 
who devote more time to coaching teachers, developing 
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the educational program, and conducting teacher evalua- 
tions are leading schools that show significant improve- 
ments over time in student math test scores (Grissom, 
Loeb, & Master, 2013). Interestingly, the frequent practice 
of instructional walk-throughs (City, Elmore, Fiarman, & 
Teitel, 2009) revealed negative associations with achieve- 
ment growth, and none of the factors related to math 
achievement were significant for reading achievement 
(Grissom et al., 2013). 

Leadership takes many forms with “distributed leader- 
ship” having received considerable attention (Spillane & 
Healey, 2010). Looking beyond the principal and admin- 
istrative team, leadership extends to others who guide, 
direct, and influence a school’s educational programs. This 
elevates the importance of considering the utility of 
teacher-leaders (York-Barr & Duke, 2004) along with the 
possibility that science leadership requires special actions 
distinct from leadership for other subjects—perhaps even 
more so when leading science in culturally and linguisti- 
cally diverse settings (Scanlan & Lopez, 2012; Stein & 
Nelson, 2003). Therefore, when investigating leadership’s 
potential impacts on student science achievement, all who 
contribute to leading the school need to be taken into 
account. Paralleling Hiebert and Morris (2012) who rec- 
ommend a focus on teaching instead of just the teachers, 
this study considers leading rather than simply the school 
leaders. 


Theoretical Framework: Social Capital 
Our efforts to develop a science-specific instrument to 
explore associations between school-level variables and 
equitable science performance were informed by James 


Coleman’s (1988) tripartite notion of social capital: the 


“wealth” of organizations is encompassed within their 
social norms, informational channels, and reciprocating 
relationships (Dika & Singh, 2002)—all of which are 
shaped by the leadership exerted within the school. Thus, 
the capacity of a school to react to demands placed on it 
and its resiliency to perturbations reflect the supply of 
social capital (Holme & Rangel, 2012). Social capital 
offers a useful ecological perspective (cf. Scholz & 
Reydon, 2008) contrasting with a school capacity or 
human capital paradigm (Odden, 2011). Smylie and Evans 
(2006) posit that using social capital as a theoretical lens 
allows educational researchers to examine relationships in 
organizational structures that may support or impede edu- 
cational change. Toward these ends, we sought to capture 
the “investments” schools make in children’s educational 
futures through schoolwide efforts at fostering organiza- 
tional trust, maintaining group norms, and preserving 
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communication networks (Daly, Moolenaar, Der- 
Martirosian, & Liou, 2014). In this regard, this study was 
grounded in Coleman’s social capital theory with the goal 
of better understanding the influences on achievement by 
elementary school students (Kahlenberg, 2001). 

With social capital as our underlying theory, we present 
a two-part study of school-level factors and student 
science achievement. First, we describe the development 
of the School Science Infrastructure (SSI) survey that 
promises to be a tool for capturing teacher perceptions of 
schoolwide factors. Second, we describe a foray into 
assessing the tool’s utility by comparing teachers’ 
responses to the SSI with student performance on the 
annual statewide science test. The study was guided by two 
research questions corresponding to the overall structure 
of this article, namely: 

1. What does the SSI capture about teachers’ views of 
organizational and leadership structures and practices, 
particularly in regard to science learning and student 
diversity? 

2. What associations exist between SSI factors and 
school performance on a statewide science test after 
accounting for student demographics? 


SSI Instrument: Development Process 

The development of the SSI followed a very deliberate 
process involving 18 months of effort consisting of dis- 
tinct phases (Table 1). First, a deliberately broad list of 
categories and items was generated, starting with an orga- 
nizational report on Chicago elementary schools (Bryk 
etal., 2010) along with an exhaustive search of the 
research literature. For example, we elaborated on “school 
climate” (to include emphasis on academics, students’ 
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Table 1 
Summary of the SSI Development Process 
Phase Purpose Participants Results 
1 Initial item Research team 200 items across 
pool 18 categories 
2 Content 13 Science Reduction of 
validation educators items to just 
under 100 
3 Teacher 131 K-8 teachers Refinement of 
validation items for 
phrasing and 
structure 
4 Pilot test #2 504 K-8 teachers EFA suggests 9 
nationally factors for 67 
items 
5 Pilot test #3. ~700K-8 CT and _ CFA leading to 
FL teachers 64-item SSI 
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sense of safety, and maintenance of order), “professional 
capacity” (to encompass professional development, 
deprivatized practices, and faculty commitment to the 
school), and “school leadership” (to incorporate instruc- 
tional, administrative, distributed functions along with 
commitments to equity). The initial pool consisted of 
nearly 200 items dispersed across 18 categories as pre- 
sented in Table 2. 

Because. of the highly ,varied ways in which science 
instruction is provided in elementary school, many teach- 
ers are not directly engaged with helping children to learn 
science (Banilower et al., 2013). With the school as the 
unit of analysis, it was important to assess schoolwide 
factors without eliminating those adults who do not teach 
science. Teachers not formally assigned to teach science 
would nonetheless be aware of the emphasis of science 
within the school’s larger mission. Rather than exclude 
such perceptions, the items were developed without pre- 
suming a respondent was responsible for providing 
science instruction. Since the survey itself focused on 
schoolwide science programming, many items transcend 
the particulars of science teaching in favor of capturing 
broader essences of science education throughout the 
building. Even though many SSI items did not specifically 
include the word “science,” respondents likely recognized 
its implied presence across all survey prompts. 

An expert content panel of 13 individuals was recruited 
from among science education researchers, classroom 
teachers, school administrators, and district science coor- 
dinators (e.g., via the National Science Education Leader- 
ship Association). Filtered through their expertise, the 
panelists conducted a content analysis: sorting each item 
into the category that it best seemed to belong, indicating 
the confidence with which they placed the item in that 
category, and then indicating the degree of alignment 
between the item and the category definition. This infor- 
mation was used to modify the prompts while simultane- 
ously reducing the number of items. 

The subsequent pilot test with 131 kindergarten through 
eighth grade (K-8) teachers surfaced additional issues, 
prompting revision of the SSI. In particular, the pattern of 
responses revealed that items proffering uncomplimentary 
views of others clustered as a separate factor. In response, 
negatively worded items were revised or deleted. Other 
items for which there were inconsistent responses were 
eliminated, leading to a revised survey that served as the 
basis for subsequent field testing. 

A national marketing company that maintains panels of 
educators for research studies was contracted to supply 
500+ K-8 teachers to participate in the next content 
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Table 2 
Initial Pool of Items 


A. Leadership within the School 


Intellectual and instructional leadership The school leadership identifies science goals, supports development 
of science teaching expertise, and displays expertise about 
ne - supporting science instruction. 
Coordinating processes and distributing responsibilities The school leader creates collaborative opportunities, maintaining 
lines of communication, and encouraging involvement in 
decision-making. 


Administrative management The school leader effectively manages the budgetary, scheduling, 
staffing, and material needs associated with the operation of the 
school. 

Leadership promoting commitments to equity The school leader acknowledges educational inequities and encourages 


efforts to reduce the associated inequities. 


B. School Climate 





Emphasis on academics The schools focus on strengthening students’ understandings of 
subject matter. 

Students’ sense of safety Students feel protected, appreciated, and successful within this school. 

Maintaining safety and order Efforts by adults to ensure that the school is welcoming, orderly, and 
safe. 


C. Professional Capacity 


Professional development The impact of professional development on the recipients. 

Deprivatized practice Forms of collaboration with colleagues to advance science teaching 
practices. 

Commitment to the school Satisfaction with and commitment to being a teacher at one’s current 
school. 


D. Ties Among Teacher/Family/Community 


Teacher direct agency Teachers communicate with families about events taking place in the 
classroom. 

Teacher proxy agency Teachers partner with families to extend curriculum-related activities 
into the home. 

Teacher collective agency Teachers incorporate community resources into instruction to advance 


student learning. 


Community support services : Familiarity with social supports that could provide services to children 
and families. 


E. Trust (in Other Teachers, in the Principal, or with the School Community) 


Competence Other teachers within the school (or the principal) possess necessary 
expertise to capably carry out responsibilities, have a reasonable 
professional repertoire, and are adept at selecting from and applying 
those in their work. 

Integrity Other teachers (or the principal) exhibit consistency between words 
and deeds, statements, and actions, and this consistency provides 
the assurance that one’s individual needs will be met and justice 
will be upheld. 

Personal regard The teacher is cared about and valued by other teachers (or the 
principal). Relationships reflect mutual goodwill with the 
expectation that one’s well-being will be protected rather than 
exploited. 


Openness The school community operates with transparency in terms of 
information and explanations, policies and decision-making, 
operations, actions, and behaviors. 


Wn 
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Table 3 
Descriptive Statistics from Confirmatory Factor Analysis (n = 661) 


Factor # of Items 


. Principal Dependability 16 
. Collaborative Teacher Learning in Science y 
. Other Teachers’ Trustworthiness 1] 
. Social Service Knowledge 

Families as Educational Partners 

. Teacher Pride and Orderly Environment 
. Principal Advocacy for Equity 

. Supports for Science Teaching 

. Shared Decision-Making 


HRuOnANAA 


Cronbach’s Alpha Inter-Item Correlates: Mean (SD) 





SD, standard deviation. 


analysis. Item responses by this population were examined 
using Exploratory Factor Analysis to uncover whether 
responses coalesced into distinct categories. Bryk et al. 
(2010) claimed “five essential supports” that determine 
which urban elementary schools are capable of supporting 
student success in language arts and mathematics. We 
maintained a more objective stance by relying on partici- 
pants’ responses to identify categories. Many items had to 
be discarded because they failed to aggregate into factors. 
The result was a manageable 67 items grouped into nine 
different factors. 

We conducted a confirmatory factor analysis (CFA) 
using responses from nearly 700 practicing K-8 teachers 
across four metropolitan school systems from Connecticut 
and Florida. The result was a 64-item survey that measures 
teacher perceptions about their schools’ organizational 
structures and leadership practices. This iterative and evi- 
dentiary process reflects the desire to situate the instru- 
ment in the larger body of research while ensuring that the 
survey was valid and reliable. 


SSI Instrument: Analysis and Results 

To determine an appropriate number of factors to 
extract, we examined participant responses via a parallel 
analysis, the minimum average partial procedure, the scree 
plot, and the magnitudes of the eigenvalues. Taking all of 
this into consideration, a nine-factor solution provided the 
best solution for the data. After further revisions to the 
items and collecting data from a fresh pool of teachers, we 
conducted a CFA using Mplus 7. Because the items were 
measured on a S-point Likert scale, we treated the 
response variables as ordinal and computed weighted least 
square parameter estimates using a diagonal weight matrix 
with mean and variance adjusted chi-square statistics that 
use a full weight matrix (Muthén & Muthén, 2013). The 
results of the CFA indicated that the nine-factor model for 
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97 .72 (.170) 
88 45 (.110) 
90 .46 (.080) 
88 .65 (.056) 
.80 50 (.126) 
90 5 Gi22) 
83 y 56°CL79) 
90 : 65 (.114) 
82 54 (.055) 
Table 4 
Means and Standard Deviations within Factor Subscales, Teacher Level 
Factor Mean Standard xn 
Response Deviation 
1. Principal Dependability 4.17 29 697 
2. Collaborative Teacher Learning 3.40 83 690 
in Science 
3. Other Teachers’ Trustworthiness 4.24 62 692 
4. Social Service Knowledge 35 LS 690 
5. Families as Educational Partners 2.85 95 692 
6. Teacher Pride and Orderly 4.16 .84 694 
Environment 
7. Principal Advocacy for Equity 4.33 .83 694 
8. Supports for Science Teaching 3.60 1.01 692 
9. Shared Decision-Making 3.78 .98 694 
Table 5 
Means and Standard Deviations within Factor Subscales, School Level 
Factor Mean Standard 7” 
Response Deviation 
1. Principal Dependability 4.07 61 23 
2. Collaborative Teacher Learning Sali 38 23 
in Science 
3. Other Teachers’ Trustworthiness 4.17 eT eS 
4. Social Service Knowledge 3.10 7; 23 
5. Families as Educational Partners Deh 54 28 
6. Teacher Pride and Orderly 3.99 50 23 
Environment 
7. Principal Advocacy for Equity 4.25 50 23 
8. Supports for Science Teaching 3.32 9 23 
9. Shared Decision-Making 57 oe) 23 





the SSI instrument provided an adequate fit to the data 
(Tables 3, 4, and 5). 

The resulting nine factors consisted of between 4 and 
16 items with correlations among factors ranging from .21 
to .93, with the highest correlation between Principal 
Dependability and Principal Advocacy for Equity. 


Volume 115 (8) 


Science From Organizational Perspectives 


Table 6 


Correlations Among Nine Factors in the SSI 
ee 


Geen es Sc ESS HO oT on THO SHO 


#1 1.00 
#2. “AT 1.00 
Ho ead ih6h 1.00 


#4 SR 2” 36. 1.00 
45" “Ba 47 Ai 93 
HoweeeD CHEE 96 


1.00 


Say igo-4o 


Hin ostS 466 » S40 ilk: E3540 761000 
Poets SOL. © 60 AO OAO. 165) «83-100 
wo eed 66863. 37 Ad FT BA BH. 1.00 





Although conceptually the correlation between Principal 
Dependability factor and the Principal Advocacy for 
Equity factor should be strong, the statistics hinted at 
potential discriminant validity issues. We conducted a 
second factor analysis that combined the responses of 
these two factors resulting in an eight-factor model. The 
corrected chi-square test indicated that the nine-factor 
solution provided a better fit to the model than did the 
eight-factor solution (chi-square = 82.79 with 8 degrees of 
freedom). The scores on the nine scales exhibited adequate 
internal consistency with Cronbach’s alphas ranging from 
.80 to .97. Correlations among the nine factors are pre- 
sented in Table 6. 

The first research question implied a degree of hesita- 
tion about whether survey items could assess elementary 
school teachers’ views of organizational structures and 
leadership practices within their workplace, with a particu- 
lar emphasis on science learning and student diversity. The 
findings offer a resounding confirmation that the SSI cap- 
tures school-level features about school infrastructure. As 
with other instruments grounded in social capital theory, 
the SSI captures both the relationships among educators 
within the school as well as networks between the school 
and external groups (Holme & Rangel, 2012; Scanlan & 
Lopez, 2012). These variants of organizational trust have 
commonly been described within social capital theory as 
“bonding” for within institution relationships (e.g., among 
the teachers) versus “bridging” for cross-institutional 
associations (e.g., between the school faculty and social 
service agency) (Nooteboom, 2007). A distinct advance of 
the SSI over other measures is the attention given to 
science-specific features of a school. This includes items 
specific to the principal’s leadership in science and the 
extent of teacher collaborations around science. 

Two rounds of factor analysis confirmed that the teacher 
responses aggregated into nine distinct factors. Further, 
the nine factors resonated with the three aspects of social 
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capital proposed by Coleman (1988). More than affirming 
social capital theory’s usefulness, the analyses brought to 
the surface sources of controversy within school effective- 
ness research (Scheurich, Goddard, Skrla, McKenzie, & 
Youngs, 2011). For example, while “trust” is typically 
portrayed as infused throughout schools (Bryk & 
Schneider, 2002), the data demonstrate that trust is not a 
unitary force but instead takes various forms (Mayer, 
Davis, & Schoorman, 1995). The Principal Dependability 
and Other Teachers’ Trustworthiness each captured “trust” 
with response patterns distinct from one another 
(Smetana, Wenner, Settlage, & McCoach, 2015). Simi- 
larly, rather than a one-dimensional view of leadership, 
teachers hold a multifaceted perspectives about the roles 
of the building principal: the managerial aspects of their 
principal’s job (within Principal Dependability) were in a 
category distinct from instructional leadership (from Sup- 
ports for Science Teaching), both of which were statisti- 
cally distinct from Principal Advocacy for Equity. Thus, 
elementary school teachers conceptualize schools as mul- 
tidimensional and view leadership as more nuanced than 
simply the role played by someone holding the title of 
principal. 


SSI and Science Test Scores: Statistical Setup 

To explore the potential utility of the SSI, we investi- 
gated statistical relationships between teachers’ SSI 
responses and students’ science achievement. The sample 
for the multilevel analyses consisted of 314 K—8 teachers 
from 28 Connecticut schools. Rather than use raw science 
test scores, we wanted to quantify school performance in a 
way that acknowledges the schools’ student demograph- 


_ics. Multiple regression made this possible by generating a 


measure called the “residual.” A school’s science residual 
indicates the degree to which students under- or 
overperformed statistical expectations while accounting 
for student demographics. A positive residual indicates 
that a school performed better than statistical projections 
would have predicted given who attends the school. Simi- 
larly, a negative residual reveals that a school performed at 
a lower level than expected relative to all schools in the 
state. To compute school science residuals, we built a 
multiple regression model using school-level demographic 
variables associated with groups underrepresented in 
college science majors and postsecondary scientific 
careers: percent of Black students, percent of Latino stu- 
dents, percent of children from families whose incomes 
qualify them for free/reduced price lunch (FRPL), and the 
percent of English language learners (ELLs). We also 
created quadratic and interaction terms for all of the 
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Table 7 \ 
Variance Components and Inter-Item Correlations (N = 314) 








Factor Between-School Within-School Intra-Class 
Variance Variance Correlation 
1. Principal Dependability 247 87 a2 
2. Collaborative Teacher Learning in Science LT .63 16 
3. Other Teachers’ Trustworthiness 054 ba 14 
4. Social Service Knowledge .067 1.30 05 
5. Families as Educational Partners 252 .64 a, 
6. Teacher Pride and Orderly Environment .183 ~? 24 
7. Principal Advocacy for Equity 162 56 23 
8. Supports for Science Teaching 216 87 24 
9. Shared Decision-Making 184 89 mi 
Table 8 
Percent Reduction in Between-School Variance with Science Residuals 
Factor Between-School Between-School Reduction in 
Variance Model 1 Variance Model 2 Between-School 
Variance 
1. Principal Dependability 247 18 27.9% 
2. Collaborative Teacher Learning in Science alt .08 27.7% 
3. Other Teachers’ Trustworthiness 054 05 0% 
4. Social Service Knowledge .067 .06 Tp 
5. Families as Educational Partners woe .16 29.6% 
6. Teacher Pride and Orderly Environment 183 .08 54.9% 
7. Principal Advocacy for Equity .162 14 15.4% 
8. Supports for Science Teaching .276 ,h9 29.6% 
9. Shared Decision-Making 184 13 27.7% 


independent variables and included them in the regression 
model. The statistically significant predictors of school 
science achievement were: (a) the percent of FRPL stu- 
dents, (b) the percent of ELLs, (c) the squared percent of 
ELLs in the school, and (d) the squared percent of black 
students in the school. This model explained 81% of the 
variance in school science achievement. 


SSI and Science Test Scores: Results 

After running a baseline empty model to estimate the 
proportion of within- and between-school variance for 
each of the nine SSI factors, we estimated a multilevel 
model in which the school-level science residual predicted 
teachers’ scores on each of the nine subscales. One notable 
finding was that the intra-class correlation coefficients 
(ICCs) indicated that 15—25% of the variances in SSI 
subscale scores were between schools on eight of the nine 
subscales (Table 7). The notable exception was Social Ser- 
vices Knowledge: the ICC for this subscale (.05) was quite 
low. One hypothesis for this low coefficient is that the four 
items associated with this subscale fall beyond the realms 
of school organization and leadership and could have con- 
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tributed to the weak correlations to the other eight SSI 
factors. This raises concerns when interpreting results 
associated with this factor. However, the magnitude of 
ICCs for the other eight subscales indicates that teachers 
who are clustered within schools are more similar to each 
other than they are to teachers in different schools. Schools 
as a whole do differ from each other in terms of the nine 
factors measured with SSI. Consequently, the SSI survey 
distinguished between schools while revealing similarities 
in perceptions by teachers within their individual schools. 

Results from the multilevel analyses revealed that 
school science residual scores could predict factor scores 
on six of the nine subscales. Table 8 displays the propor- 
tion reduction for between-school variance after adding 
the science residual score to the multilevel model. Adding 
the science residual explained 54.9% of the between- 
school variance in the Teacher Pride and Orderly Environ- 
ment subscale, and it explained at least 20% of the 
between-school variance in five of the other subscales. 
Simply stated: School performance on the statewide 
science test predicted teachers’ perceptions of organiza- 
tion and leadership factors across six SSI subscales. 
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Discussion 

This study builds on prior work in other subject areas 
showing that the practices by school leaders and the orga- 
nizational structures of a school have significant influ- 
ences on student achievement (Wahlstrom & Louis, 2008). 
To date, the potentially unique aspects of science educa- 
tion have received little attention within studies of school 
organization and leadership. The presence of “achieve- 
ment gaps” (i.e., otherwise similarly populated schools 
producing varied science test results) along with indica- 
tions that disparities associated with student ethnicities 
result from complex dynamic processes (Mickelson, 
2003) provoked the need for a conceptual shift in our 
framing of elementary science education. School effec- 
tiveness research has demonstrated strong associations 
between school-level factors and student academic perfor- 
mance in math and reading (Hallinger & Heck, 2011). 
Rather than be lured into the belief that successful schools 
in urban environments are a consequence of super-heroic 
leaders (York-Barr & Duke, 2004) we adopted a more 
expansive view that gave due consideration to the school 
as an ecological organization (March & Simon, 1993). . 

Grounded in school effectiveness research and social 
capital theory, the instrument reported upon here is a valid 
and reliable tool supportive of meso-level investigations of 
school factors contributing to variability in science 
achievement. While we began with a sizable pool of items, 
the combination of expert content validation and a series 
of field tests resulted in a compact and robust instrument. 
In regard to the first research question, the 64-Likert scale 
items of the SSI generated scores for nine statistically 
distinct factors. For the second research question, when 
SSI responses by teachers within a single state were ‘com- 
pared to school science residuals, we found that multiple 
SSI factors (but not all) were: significantly related. While 
this is not definitive given the restricted scope of the study, 
it nonetheless suggests that the SSI is a useful tool for 
supporting further studies of school effectiveness specific 
to science achievement. 

The SSI instrument is grounded in social capital theory 
(Coleman, 1988) and constituted by three interrelated fea- 
tures: reciprocating relationships, informational channels, 
and organizational norms. Reciprocating relationships are 
measured via Principal Dependability and Other Teachers 
Trustworthiness. Informational channels appear within 
Collaborative Teacher Learning (e.g., feedback from peers 
and idea sharing), Principal Advocacy for Equity (e.g., 
messaging and advocating), and Collaborative Teacher 
Learning in Science (via coaching, meeting, planning, 
etc.). Finally, organizational norms reside within Shared 
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Decision Making, Families as Educational Partners, and 
Teacher Pride and Orderly Environment. That Social Ser- 
vices Knowledge does not clearly fit within any of the three 
forms of social capital could explain its poor showing 
relative to the other SSI factors. Still under consideration is 
a compelling mechanism through which social capital pro- 
motes more equitable science achievement. Potential 
explanations range from social capital as a commodity that 
collectively enriches the organization (Spillane, Kim, & 
Frank, 2012) to the possibility that social capital supports 
organizational alignment (Penuel et al., 2010). In short, the 
SSI reveals itself to be a reliable tool for studying science at 
the school-level—questions linger about the exact mecha- 
nisms through which social capital enhances student 
science performance. 

The next phase of research will expand upon the work 
reported here to include a variety of qualitative approaches 
as well as examining teacher advice networks (Lee, 2014). 
As with any research tool or methodology, there are cau- 
tionary limitations, foremost of which is that the data 
provided via the SSI are teacher anonymous self-reports. 
In addition, there are reasonable concerns about represent- 
ing the complexities of schools using quantitative descrip- 
tions. However, when used to complement qualitative 
examinations of a school, the SSI provides a useful metric. 
At this stage, the use of the SSI has been restricted to 
teachers working with students in pre-high school settings. 
We suspect that in schools departmentalized by subject 
matter, the organizational dynamics might produce differ- 
ent results. Furthermore, the development of the instru- 
ment focused solely on teachers in the United States and 
would benefit by being applied in other countries. We 


perceive these as important next steps for the SSI and 


nominate these as questions deserving of continued 
pursuit. In the meantime, our research continues to benefit 
from Coleman’s (1988) view that educational institutions 
are responsible for supporting children’s achievement 
rather than placing blame for underachievement on fail- 
ures alleged to be rooted in biological, community, ethnic, 
or familial shortcomings. Finally, we remain optimistic 
that adopting an ecological perspective toward school 
science, informed by using the SSI to measure organiza- 
tional and leadership factors, we can subsequently devise 
interventions to assist with reducing the science achieve- 
ment disparities that have been heretofore resistant to 
change. 
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This paper presents the results of a multi-method study examining elementary students with high self-reported levels 
of mathematics motivation. Second- through fifth-grade students at a Title One school in the southeastern United States 
completed the Elementary Mathematics Motivation Instrument (EMMI), which examines levels of mathematics moti- 
vation across three subscales: (a) Math Anxiety, (b) Self-Efficacy, and (c) Value of Math. Results from this quantitative 
phase were used to identify a sample for a qualitative phase examining how students who report high levels of 
motivation perceive mathematics. The resulting qualitative phase utilized a phenomenological design to explore 
mathematics motivation for a particular set of students in a fifth-grade setting. Findings indicate that elementary 
students with high mathematics motivation value mathematics as a present and future oriented discipline and value 
teachers that deemphasize testing as a measure of success. 


Motivation has been identified as a factor that is essen- 
tial for elementary students to be successful when engag- 
ing in mathematical tasks. Positive student perceptions 
regarding the ability to solve mathematical tasks, as well 
as the value of these tasks, are necessary to support student 
understanding of mathematical concepts (National 
Research Council [NRC], 2001). Research indicates that 
beliefs about mathematics are developed early in life and 
that once established, are difficult to change. If students 
display low levels of motivation for mathematics at a 
young age, these perceptions of ability and value are likely 
to remain constant as they continue through their aca- 
demic career (Gottfried, Marcoulides, Gottfried, Oliver, & 
Guerin, 2007). The relationship between motivation and 
achievement and the need to establish positive dispositions 
at an early age provide a rationale to examine factors that 
may influence mathematics motivation for students in 
elementary grades. 

The present study explores characteristics of high 
motivation in children from kindergarten through fifth 
grade. A sequential, multi-method design was utilized to 
answer two central research questions: How do highly 
motivated elementary mathematics students describe and 
experience mathematics and how can teachers support 
student motivation for mathematics? The overall purpose 
of the study was to identify student, teacher, and class- 
room factors that have the potential to influence motiva- 
tion in a positive manner in an effort to move toward 
developing a motivation framework in elementary 
mathematics. 
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A wealth of literature supports the significance of moti- 
vation on performance and various affective factors for 
students. Motivation has the potential to predict student 
performance (Murayama, Pekrun, Lichtenfeld, & vom 
Hofe, 2013), self-regulation, and effective goal setting 
(Crede & Kuncel, 2008), persistence (Tulis & Fulmer, 
2013), as well as other positive predictors of student learn- 
ing (Federici & Skaalvik, 2013). Several theories have 
been developed to examine student motivation such as the 
expectancy—value model that stresses the importance of 
goals and value in making decisions. Using this model, 
researchers posit that self-beliefs and task-beliefs have a 
reciprocal influence on choices (Eccles, 2011). 

In addition, the expectancy—value model has shown that 
an increase in motivation can also increase the amount of 
attention and energy directed toward activities that can 
positively impact student learning (Eccles & Wigfield, 
2002). Expectancy-value theory posits that motivation is a 
function of an individual’s expectancy for success for a 
given task and the individual’s value for the task (Eccles & 
Wigfield, 2002). This theory grew from the early work of 
Atkinson (1957) and was developed throughout the 1960s 
and 1970s (Battle, 1965; Crandall, 1969) and is recently 
associated with the work of Eccles and Wigfield (2002). 
Within this model, expectancies for success and task value 
are the two primary constructs related to an individual’s 
motivation. Interestingly, these constructs have been 
studied together and in isolation; research indicates that 
task value is often predictive of an individual’s choices or 
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decisions while expectancies for success are more predic- 
tive of performance (Wigfield & Eccles, 2002). 

Increased motivation can support student persistence 
through difficult tasks and continue to challenge them in a 
particular content area. Bandura (1997) discussed the sig- 
nificance of risk taking and persisting in his theory of 
self-efficacy; he noted that self-efficacy can be predictive 
of an individual’s motivation, affect, and behavior. In fact, 
Bandura (1997) postulated that students’ self-efficacy for 
a domain may be a better predictor for their achievement 
in that specific content area than objective assessments. 
For example, students who are efficacious about their 
ability to learn mathematics are more likely to attempt 
challenging tasks, persist at those tasks, and make positive 
attributions for both success and failure (Bandura, 1989, 
1997). The opposite is true of students with low self- 
efficacy for learning mathematics. 

Other theories of motivation, such as self-determination 
theory, have been used to predict motivation outcome vari- 
ables like investment in learning activities (Kasser & 
Ryan, 1996), persistence, and level of achievement 
(Reeve, Deci, & Ryan, 2004). By incorporating various 
theories of motivation, a more comprehensive framework 
for motivation can be put forward. This study builds on the 
theories of expectancy—value and self-efficacy, in addition 
to considering math anxiety, which has been linked to 
motivation in prior research (e.g., Zakaria & Nordin, 
2008), in order to formulate a theoretical framework for 
mathematics motivation. 

Although a motivation framework lays out a lens for 
examining student motivation, it is important to also con- 
sider how motivation is linked to what is occurring in the 
classroom. Students who display high levels of motivation 
often use critical thinking skills when problem solving and 
more sophisticated metacognitive strategies for retaining 
and utilizing new information (Pintrich & Schunk, 2002). 
The connection between motivation and students’ specific 
strategies for learning highlights the importance of inves- 
tigating motivation toward mathematics separately from 
other content areas. Previous research indicates that moti- 
vation is context specific and that mathematics motivation 
is connected to mathematics achievement (Wigfield & 
Eccles, 1994). 

Motivation can be predictive of study skills and student 
engagement, which positively correlate with mathematics 
achievement (DiPerna, Volpe, & Elliott, 2005). Student 
factors such as self-efficacy, anxiety, and task value have 
also been related to student academic performance 
(Shores & Shannon, 2007). In addition, these factors have 
influenced student regulatory strategies, both adaptive and 
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maladaptive, while learning mathematics (Cleary & Chen, 
2009). Finally, research indicates that motivation can have 
an impact on student participation in a mathematics class- 
room (Jansen, 2008). These studies support the signifi- 
cance of student motivation in a mathematics classroom, 
providing evidence that motivation is a critical construct 
with ultimate connections to student achievement in math- 
ematics. However, previous research has tended to 
examine the implications of motivation rather than provide 
guidance as to characteristics or methods that may lead to 
an increase in student motivation for mathematics. 


Instructional Practices and Motivation 

The National Council of Teachers of Mathematics 
(NCTM, 2014) recognized the necessity for motivational 
strategies to be included in the instructional practices of 
teachers. As NCTM stated, teachers’ goals are to “ensure 
that all students become confident” (NCTM, 2014, p. 109) 
and “promote positive dispositions toward the student of 
mathematics” (NCTM, 2014, p. 114-115). The teacher’s 
role was made even more explicit by Pintrich (2003) when 
he summarized five subscales from existing motivation 
literature that had direct implications for the design of 
instructional practices to motivate students: (a) “adaptive 
self-efficacy and competence beliefs,” (b) “adaptive attri- 
butions and control beliefs,” (c) “higher levels of interest,” 
(d) “higher levels of value,’ and (e) “goals” (Pintrich, 
2003, p. 672). 

Research focusing on using instructional technology is 
one way that motivation has been related to instructional 
practice. Positive correlations have been found between 


classroom instructional technologies and student motiva- 


tion in upper elementary students (Torff & Tirotta, 2010). 
However, these effects were found to be dependent on the 
teachers’ support and beliefs about technology. This 
finding supports evidence from other research that teacher 
beliefs impact components of student motivation such as 
mastery and performance goals (Muis & Foy, 2010). Muis 
and Foy (2010) concluded that further research needed to 
involve not only observing classroom instruction but also 
interviewing teachers and students to better understand the 
complex nature of investigating these affective relation- 
ships. There is a need for additional research to further 
explore how instructional practices influence students’ 
motivation and achievement (Byrnes & Wasik, 2009). 
Further, there is substantial evidence that motivation is not 
generalizable to all students and that considering other 
external factors might be necessary when examining stu- 
dents’ motivation (Murphy et al., 2010). 
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By further investigating how students describe their 
experiences with mathematics, researchers might be better 
able to understand students’ motivation and connections to 
instructional practices and teacher factors. In the current 
study, we examine factors contributing to the formation of 
student mathematics motivation, particularly high motiva- 
tion. By sampling elementary students who report high 
levels of motivation, we are able to isolate and examine the 
external and internal factors contributing to these results 
and therefore begin to make sense of how to positively 
affect student motivation in mathematics. 


Methods 

The central research questions guiding this study were 
(a) How do highly motivated elementary mathematics stu- 
dents describe and experience mathematics and (b) How 
can teachers support student motivation for mathematics? 
For the purposes of this study, the research team focused 
on characteristics of highly motivated students rather than 
examining low levels of motivation. There is already an 
abundance of research identifying motivational risk 
factors, such as mathematics anxiety, instructional prac- 
tice, and parental influence. The intent was to determine 
how teachers can capitalize upon or encourage character- 
istics of highly motivated students in their own mathemat- 
ics instruction. The research questions were examined 
using a multi-method design where quantitative data were 
collected and used to inform the sampling approach for a 
subsequent qualitative investigation of student perceptions 
about mathematics motivation. 
Participants 

Participants in Phase One were second- through fifth- 
grade students (NV = 288) in 16 classes at a Title One 
magnet elementary school, with an emphasis on math- 
ematics and science education, located in the southeastern 
region of the United States. The research team chose this 
school because of the diversity of students, the magnet 
emphasis, and because the curriculum and instructional 
practice of the majority of teachers reflected progressive 
practices in mathematics (NCTM, 2014) as determined 
through classroom observations and prior work with 
student teachers in the setting. These characteristics were 
important because the intent of the study was to identify a 
variety of students with high levels of mathematics moti- 
vation. The research team hypothesized that the use of 
effective instruction in mathematics could potentially 
impact student motivation. 

The research team used the survey results from Phase 
One to recruit participants for Phase Two. The research 
team originally intended to sample highly motivated stu- 


394 


dents from every participating grade level. However, 
results from Phase One showed higher levels of motivation 
for all students at the fifth-grade level. These findings were 
particularly interesting because research suggests stu- 
dents’ motivation levels decline as they progress through 
elementary grades and into middle grades (Pajares, 
Britner, & Valiante, 2000). The research team wanted to 
further explore why fifth-grade students in this setting 
were displaying such high levels of mathematics motiva- 
tion. Therefore, the original sampling plan was dynami- 
cally adjusted to focus on these highly motivated students 
at the fifth-grade level (n = 20). 
Instrumentation 

The Elementary Mathematics Motivation Inventory 
(EMMI) is a 17-item self-report instrument that was devel- 
oped to measure motivational constructs in children. Items 
contained positively or negatively phrased statements of 
disposition toward math, prompting students to respond 
with a four-point Likert-type scale; “just like me (4),” 
“sort of like me (3),” “not really like me (2),” and “defi- 
nitely not like me (1).” Items in the instrument were 
written to prompt student judgments related to (a) anxiety 
(Shipman & Shipman, 1985), (b) interest (Hidi & 
Anderson, 1992), (c) task value (Wigfield & Eccles, 
2002), (d) self-efficacy (Bandura, 1997), and (e) goal ori- 
entation (Elliott, McGregor, & Gable, 1999). The survey 
was determined to have a Flesch—Kincaid reading level of 
3.1. The following subscales are represented within the 
EMMI: (a) Math Anxiety, (b) Self-Efficacy, and (c) Value 
of Math. In a validation study (V= 1,018), the EMMI scale 
showed high reliability, with Cronbach’s alpha at .833 for 
the 17-item scale and average Cronbach’s alpha of .812 for 
the three subscales (Linder & Smart, 2010). 
Phase One 

Phase One examined levels of student motivation in 
mathematics across the elementary grades. This quantita- 
tive phase utilized a survey design, in which elementary 
students (n = 288) completed the EMMI (Linder & Smart, 
2010), to identify a sample of students who reported high 
levels of motivation for mathematics. The three-member 
research team administered the surveys in person by vis- 
iting all second- through fifth-grade classrooms over a 
two-day period in spring of 2010. Survey items were read 
aloud to participants in all grade levels to control for 
reading ability. The teacher remained present during 
survey implementation to alleviate student testing fears 
and to assist students who required extra aid. 

Data were collected for participant selection purposes; 
therefore analyses were limited to comparisons of levels of 
motivation. The researchers first computed the reliability 
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Table 1 
Semi-Structured Interview Protocol 


esp umes et neces: Sse sew need eee enee iss nce seo amen eae marine teatime 


Participants: 


Questions 
1. Do you enjoy math? Follow-up: Why/Why not? 


Date: 


Time: 


If yes: Is math a subject that you have always enjoyed or have you recently begun to enjoy it? 
2. Do you feel like you are good at math? Follow-up: What makes you feel that way? 
3. Do you think that math is important? Follow-up: Why do you think this? 
If clarification is needed: How does math impact your life or the lives of people you know? Will you use math in the future? 


How? 


4. Describe a typical math lesson. Follow-ups: What do you do as a student? What does your teacher do? What do you like 
about math lessons? What do you dislike about math lessons? Describe how you would change lessons to make them more 


enjoyable. 


5. What do you do when you have a difficult problem or question in math? Follow ups: If you don’t understand something in 
math class, what do you do? How do you feel when you get a problem wrong in math class? 

6. Does your teacher think that math is important? Follow-up: How do you know? 

7. Can you describe your math teacher for me? Follow-ups: What are your favorite things about your math teacher? Can you 
give me an example? What are some things you wish you could change about your math teacher? Can you give me an 


example? 


8. Do you feel comfortable asking your teacher for help in math? Follow-up: Why/Why not? What does she do that makes you 


feel that way? 


9. How do you feel about the way your teacher talks to you in math class? 
10. If you have something important to say, will your math teacher listen to you? 
Follow-up: How does that make you feel about math? How does that make you feel about your teacher? 





coefficient to establish the internal consistency of the 
overall survey instrument and then again for each of the 
three motivational subscales identified through principle 
components factor analysis in a previous validation study 
(Math Anxiety, Self-efficacy, and Value of Math) (Linder 
& Smart, 2010). Following this examination, the research- 
ers conducted f-tests and examined descriptive statistics 
for all participants to determine if their responses were 
significantly different than the midpoint on the EMMI 
scale (34.5). Next, the research team used analyses of 
variance (ANOVAs) to compare levels of motivation 
established from the overall scale for students in different 
grade levels and with different teachers. This was followed 
by an investigation of the three subscales within the EMMI 
scale by conducting multivariate (MANOVAs) to examine 
the relationships between these subscales and the two units 
of analysis (grade level and teacher). Finally, because 
fifth-grade students emerged as highly motivated and the 
research team made the decision to focus on this group in 
Phase Two rather than sample from grades two through 
five, the research team examined descriptive statistics to 
determine which students displayed the highest levels of 
motivation across classes at the fifth-grade level. 
Phase Two 

Phase Two of this study explored how students who 
reported significantly higher levels of motivation experi- 
ence mathematics. As described above, data collected in 
Phase One were used to select a sample of participants for 
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Phase Two who reported high motivation in mathematics. 
Findings from Phase One indicated that students in all 
fifth-grade classes displayed higher levels of motivation 
than students in the second, third, and fourth grades. These 
results are described in more detail below. The research 
team interviewed a sample of students from each of the 
fifth-grade classes. More specifically, a total of 20 partici- 
pants, or approximately 30% of the fifth-grade sample 
from Phase One, participated in Phase Two. Student 
descriptions of mathematics were examined through a 
phenomenological lens (Moustakas, 1994) by conducting 
semi-structured interviews with participants to understand 
how they experienced mathematics and then triangulating 
the interview results with videos of classroom instruction 
to determine if student perceptions were consistent with 
established classroom practices. 

Interviews were conducted in pairs to increase comfort 
level in students and to facilitate possible dialogue 
between students concerning mathematics. A protocol 
used in each interview is included in Table 1. Each inter- 
view lasted approximately 20 to 30 minutes, a typical 
length for this age group (Seidman, 2006), and was tran- 
scribed by a member of the research team. In addition, the 
research team conducted 12 scheduled classroom visits 
(three for each of the four classes) where mathematics 
instruction was observed. 

Data were analyzed in a phenomenological fashion 
(Moustakas, 1994) by identifying initial meaning units in 
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the interview data and clustering the units in themes occur- 
ring across participants. In an effort to increase trustwor- 
thiness, the researchers conducted the first round of data 
analysis individually and then worked together to identify 
themes related to student motivation. Individual analyses 
resulting in agreement in over 95% of the meaning units 
identified across interviews. The research team discussed 
inconsistencies and came to an overall agreement about 
each piece of data. 

In order to triangulate the student interview data, 
researchers examined videos and transcripts of mathemat- 
ics instruction from multiple classroom observations (NV = 
12). These classroom observations provided another 
context to examine alignment between students’ percep- 
tions of teacher practices in mathematics and actual 
observed instructional practices. The school of focus used 
a subject-centered model where teachers in second 
through fifth grade are specialists in one area of the cur- 
riculum. Using a phenomenological approach allows the 
researcher to determine perceived truths among groups in 
terms of how they interpret experiences (Moustakas, 
1994). If the themes emerging from the interviews were 
consistent with what was recorded during observations, it 
is likely that the observations were capturing a represen- 
tational picture of instructional practice. The research 
team analyzed the classroom observational data in the 
same manner as the interview data. The resulting themes 
were then compared to those emerging from the interview 
data. 


Results 

Phase One 

Reliability. Reliability for the EMMI in the present 
study was estimated by computing the Cronbach’s alpha 
for the overall*scale (.849). Further, an examination of 
“Cronbach’s Alpha if Item Deleted” suggested that all 
items should be retained. In addition, the Cronbach’s 
alpha was computed for each of the three subscales previ- 
ously identified through principal component analysis 
(PCA): (a) Math Anxiety (.655); (b) Self-Efficacy (.805); 
and (c) Value of Math (.687). The validation study (Linder 
& Smart, 2010) of the EMMI instrument yielded higher 
results for reliability for individual subscales; however, the 
sample size in that study (N = 1,018) was much higher 
than the present sample (7 = 288). The moderate to high 
levels of reliability for individual subscales warrant their 
use in analysis in addition to examining the overall scale. 

Frequencies and f-tests. 7-tests were conducted to 
determine if participants displayed significantly higher 
levels of motivation than the midpoint of the summative 


396 


scale and the midpoints of the scale for each subscale 
(Math Anxiety, Self-efficacy, and Value of Math). The 
one sample t-test for the overall scale indicated that stu- 
dents in this setting agreed with the EMMI items to a 
significant extent #272) = 10.006, p < .001. The effect 
size d of .98 indicates a large effect. When examining 
frequencies of student responses, 76% of students across 
grade levels responded with moderate to high levels of 
mathematics motivation (ranging from “sort of like me” 
with a score of 34 to 51 on the summative scale to “just 
like me” with a score of 51 to 68 on the summative 
scale) with 49% of these students responding in the high 
range of mathematics motivation. The one sample f-tests 
for two out of the three subscales were significant: Self- 
efficacy, 7(281) = 12.968, p < .001 and Value of Math, 
t(276) = 16.605, p < .001, with large effect sizes of d of 
.77 and d of .98, respectively. Frequencies of student 
responses on items for each of these subscales indicated 
that 85% of students across grade levels responded with 
moderate to high levels of motivation in terms of value 
of mathematics. Of this 85%, 83% responded with high 
levels of value toward mathematics. Eighty-two percent 
of students across grade levels responded with moderate 
to high levels of motivation in terms of self-efficacy. Of 
this 82%, 63% responded with high levels of self- 
efficacy toward mathematics. However, only 52% of stu- 
dents responded with moderate to low levels of math 
anxiety and of this 52%, 65% responded with low levels 
of math anxiety. 

These results indicate that overall, the majority of stu- 
dents in these settings displayed a fairly high level of 
mathematics motivation, specifically self-efficacy and 
value of mathematics. When examining frequencies 
between grade levels, students at the fifth grade level 
responded with moderate to high levels of motivation at a 
greater rate (86%) than students in second (68%), third 
(75%), and fourth (66%) grade. These differences 
prompted the research team to explore if significant dif- 
ferences occurred between grade levels or individual 
teachers. 

Analysis of variance. One-way analyses of variance 
were conducted to evaluate the relationship between 
student motivation for mathematics and each of the fol- 
lowing units of analysis: (a) grade level and (b) teacher. 
The dependent variable for each analysis was the 
summative score for the overall EMMI scale. There were 
no significant differences between students at different 
grade levels or with different teachers when examining the 
overall scale, so the research team then examined the indi- 
vidual subscales within the EMMI scale. 
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Multivariate analysis of variance. Following the 
analysis of the overall EMMI scale, the research team 
examined each subscale (Math Anxiety, Self-Efficacy, and 
Value of Math). A series of one-way MANOVAs were 
conducted to evaluate the relationship between two units 
of analysis (grade level and teacher) and the three 
subscales in the EMMI scale. The independent variable for 
each MANOVA was the unit of focus (grade level or 
teacher) and the dependent variables were the three 
subscales. The researchers first conducted MANOVAs to 
control for the Type I error inflation that could occur from 
conducting individual analyses of variances (ANOVAs). 
The alpha for each test was set at .05. Significant differ- 
ences were identified both between grade levels and 
between teachers: Grade (Wilks’s A = .85, F[3, 268] = 
4.904, p < .01), Teacher (Wilks’s A = .69, F[45, 761] = 
2.27, p < .01).The multivariate n2 for Grade was .05 and 
for Teacher was .12, indicating a small effect for each test. 

ANOVAs on individual dependent variables were con- 
ducted as follow-up tests to the MANOVAs. The alpha for 
each test was .05. Statistically significant findings are 
described below. 

The ANOVA for grade level and self-efficacy was sig- 
nificant (F[3, 270] = 6.80, p < .001). Follow-up tests were 
conducted to evaluate pairwise differences among the 
means. Levene’s Statistic revealed equal variances, result- 
ing in the use of Bonferroni’s post hoc comparison. Fifth 
grade students reported significantly higher levels of self- 
efficacy for mathematics than students in second, third, 
and fourth grades. 

The ANOVA for teacher and self-efficacy was also sig- 
nificant (F[15, 258] = 3.14, p < .001). In addition, the 
ANOVA for teacher and math anxiety was significant 
(F[15, 258] = 1.98, p < .05). However, post hoc results did 
not indicate significant differences among teachers in 
terms of students’ levels of self-efficacy or math anxiety. 

Based on these results, students at the fifth-grade level 
seemed to display higher levels of mathematics motiva- 
tion, specifically self-efficacy, than students in other 
grades. The overall high levels and increased frequencies 
of these levels of motivation at the fifth-grade level 
prompted the research team to further explore how these 
students perceived mathematics. 

Identifying participants. Once the research team 
established that a study of fifth-grade students was war- 
ranted based on the EMMI results, they examined the 
survey data further to identify a participant pool for Phase 
Two. The overall research questions prompted researchers 
to limit the participant pool to those who displayed the 
highest levels of motivation across the four classes rather 
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than taking a random sample from all who were willing to 
participate. Out of 62 fifth-grade students, 53% or 85.5% 
responded with motivation levels in the moderate to high 
range. These students were spread across the four classes 
with 8 students in Classroom A, 15 in Classroom B, 17 in 
Classroom C, and 13 in Classroom D. Based on these 
results, the research team recruited five students from each 
of the fifth-grade classes who displayed the highest levels 
of motivation on the EMMI. 
Phase Two 

Six overarching themes emerged in Phase Two that offer 
possible explanations for these students’ high reported 
motivation for mathematics: (a) role of the teacher; (b) 
present value of math; (c) future value of math; (d) role of 
achievement; (e) classroom environment; and (f) connec- 
tions. These themes are discussed below and interview 
excerpts are included in Table 2 to provide illustrative 
examples. The research team also determined how often 
themes were identified across all Phase Two participants. 
These findings are displayed in Table 3. To clarify, one- 
fifth grade teacher, identified as Ms. Phillips, is the math- 


- ematics instructor for all fifth-grade students. During this 


study, Ms. Phillips was finishing her dissertation for a 
doctorate in education. She had been teaching for over 20 
years, and although she did not have a specialization or a 
degree in mathematics or mathematics education, she 
reported that she enjoyed doing and teaching mathematics. 

Themes emerging from student interviews. 

Role of the teacher. Students described their teacher 
along three main subthemes: (a) Instructional practices; 
(b) Teacher’s perceived value of mathematics; and (c) 
Interpersonal support. Students often mentioned the teach- 
er’s use of group pairings, questioning, and discussion as 
positive characteristics of her instructional practice. 
However, while students expressed value in tasks that were 
hands-on and required a variety of strategies to solve, they 
often viewed their role as passive while the teacher was 
active during lessons. Students often described instances 
where the teacher would verbally communicate the impor- 
tance of mathematics for both present and future goals. 
They also expressed how much she enjoyed doing math- 
ematics and how excited she acted during lessons. Stu- 
dents described their teacher as caring, kind, funny, 
patient, and always willing to help during lessons. Stu- 
dents focused primarily on their teacher’s helpfulness 
during instruction and expressed a high level of comfort in 
interacting with their teacher, especially when seeking 
help for difficult math topics. 

Present value of mathematics. Many students indi- 
cated a high level of present value for mathematics by 
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Table 2 
Examples of Themes from Interview Transcripts 


Major Themes 


Cee ee ee ee 
° “She tries to put like someone who’s really good at math in a group with some of the ones 


Role of the teacher 


that really need help in math. Two people, two really smart people in math in a group with 
some of the people who need help in math. That’s what she does.” (Interview 3) 


“It (Math) makes me feel good because I know that I have a teacher that listens and helps 


us with the questions we have.” (Interview 4) 

* “She makes math fun, we get to play games. Last time we got to do the little chemicals 
with the soda” “Oh I know what she’s talking about. She’s talking about with the pints and 
gallons and how we had to put them in order and all that stuff”’ (Interview 6) 

Present value of mathematics * “You get to answer more difficult problems, but then again you get to find new ways to 
answer the different problems. Like for instance, if you go, if you learn something new then 
you go off to another place, new school or new college, whatever you’re going to go, you 
can just go back and just remember the thought and the different formulas to get that 


answer.” (Interview 10) 


“Like art, because you have to measure a distance, I mean like how big something is. You 


have to use a ruler just to figure out how big something is.” (Interview 4) 


“Well, we go grocery shopping and clothes shopping. We need math because we have to 


add up all the prices to be sure that you do have enough for it. Me and my cousin play a 
game where we have to figure out how much it will be. Because it can be 50% off a shirt 
and everything.” (Interview 3) 


Future value of mathematics 


“Well, um, maybe when I’m a paleontologist, I would like, predict how long the bug will be 


and how many bones there are.” (Interview 10) 


“I’m going to be a chef when I grow up and you know you got to tell about the pints and 


quarts and cups.” (Interview 6) 


“T want to be a fashion designer, so I have to tell how much I will have to make in stores 


and if they are big or little.” (Interview 9) 
Role of achievement * “She lets us redo our test. She lets us think about it instead of other teachers just rushing 
us. And she just, she helps us during tests sometimes.” (Interview 1) 
* “On the test, we might go to her and ask her, ‘we don’t really get this answer, can you help 
me out?’ She basically just helps us do it or she, or she explains how to do it and we just go 
back to our seats and do it.” (Interview 4) 
* “When we do tests, when people not done (sic), she be like, ‘I’m going to give you an extra 


999 


20 minutes. 


(Interview 6) 
“Like say you’re on a test... 


. when you are finished, she’ll grade your paper. She’ll look 


over your test. She’ll send you back to your seat (with the test) but she’ll say just keep 
trying and don’t give up.” (Interview 8) 


Classroom environment 


“Like she doesn’t just stay there and just tell us all the answers. She’s like let’s work 


together and come up with an answer to a problem” (Interview 8). 
¢ “And in her class, it’s never like there is a wrong, there is a right or wrong answer in her 


class” (Interview 2). 


», ¢ “Some problems can be like really, really hard that I can’t figure out by myself. Like I can 
go ask a student that got it right and ask for help.” (Interview 4) 


Connections 


(Interview 6) 


“She gives me choices.” (Interview 6) 
“Well, it’s like she teaches us about real life and stuff, real life things.” (Interview 1) 
“Tt’s just fun and then she like makes it easy by putting examples of how we use it.” 


* “She applies it to real life things .. . like with geometry, like with structures, how they need 
to be built. Like construction. How they need to be built just to stand.” (Interview 8) 





describing it as an enjoyable or fun subject. These students 
valued a challenge in mathematics lessons and thrived on 
attempting tasks independently before receiving direct 
help from the teacher. This independence was supported 
by students’ problem-solving strategies. When asked what 
they would do if confronted with a difficult mathematics 
problem, students described strategies such as working 
with their peers or breaking the problem into manageable 
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parts to find ways to solve it correctly. In addition, students 
described the need to keep trying to find ways to solve a 
problem that was difficult, indicating a high level of per- 
sistence for completing tasks. 

Future value of mathematics. In terms of utility 
value, students tended to view mathematics as a future- 
oriented discipline. For example, students described its 
importance in college and careers. Students tended to view 
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Table 3 
Quantitized Data from Phenomenological Analysis 


Themes and Subthemes Number of Percentage of 


Participants . Participants 
Role of the teacher 
Instructional practices 18 90% 
Teacher’s perceived value of 11 55% 
mathematics 
Interpersonal support 20 100% 
Present value of mathematics 
Challenging and fun 17 85% 
Problem solvers 14 70% 
Persistence 11 55% 
Future value of mathematics 
Career oriented 16 80% 
Role of achievement 
Priority on grades and testing 13 - 65% 
Teacher’s lack of emphasis on 10 50% 
testing 
Classroom environment 
Discourse 16 80% 
View of mistakes a 25% 
View of competition 10 50% 
Connections 
Teacher’s priority placed on 1 85% 
connections 


mathematics as applicable primarily later in their lives 
when they would be required to use their math skills in the 
context of their future career. When prompted, students 
articulated present utility of mathematics, such as purchas- 
ing items while shopping or using measurements in 
cooking were described. However, students’ initial 
responses about the usefulness of mathematics revolved 
around future uses in potential careers such as interior 
design, astronomy, and medicine. 

Role of achievement. Students tended to equate 
success in mathematics with receiving good grades. Since 
most students who participated in the interviews had expe- 
rienced academic success in mathematics, this success had 
served to reinforce their value of mathematics as an impor- 
tant subject. However, there were students included in the 
interviews with lower levels of academic success 
(described as having a C or lower) and still displayed high 
levels of mathematics motivation, indicating that achieve- 
ment is not necessarily a predictor of motivation. 

Students also mentioned both the standardized tests 
given periodically throughout the year and the culminating 
state test as an indicator of mathematics success. However, 
students described instances when Ms. Phillips would alle- 
viate stress related to testing by helping them with 
answers, letting them take the test home, or working 
through problems as a group. The students viewed Ms. 
Phillips as placing a low priority on testing in the 
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mathematics classroom, which they stated was very dif- 
ferent from teachers in the past. 

Classroom environment. Students emphasized their 
ability to work with each other and engage in discussions 
during lessons as a positive attribute toward the class- 
room environment. They also described their level of 
comfort in engaging in discourse with Ms. Phillips 
whether it was regarding questions about a task or alter- 
native methods of doing a task. When prompted about 
how they reacted to making mistakes during mathematics 
lessons, students described the need to figure out their 
mistakes or to help each other during lessons. Healthy 
competition was valued by some students, for instance, 
trying to finish a task before another group or find a 
different way to solve the problem. However, often stu- 
dents said they did not feel the need to compete with 
their classmates because they could learn from each 
other, indicating a high level of self-confidence and a 
sense of trust among their peers. 

Connections. When asked to describe a typical math- 
ematics lesson or to describe the characteristics of their 
mathematics lessons that they most enjoyed, students 
would identify instances when Ms. Phillips would ask 
them to make real-world connections to the mathematical 
content they were learning. Students described instances 
where making these connections make mathematics “fun” 
or “easy.” They described instances where Ms. Phillips 
intentionally sought out connections within a lesson and 
instances where they were able to talk to their peers and 
made connections on their own. 

Themes emerging from _ classroom _ observa- 
tions. While the student interviews provided a wealth of 
information about perceptions of mathematics and math- 
ematics motivation, it was unclear if the student responses 
related to classroom instruction (in particular, those cat- 
egorized under the Role of the Teacher, Role of Achieve- 
ment, Classroom Environment, and Connections) were 
representative of actual practice. For this reason, the 
research team further explored how Ms. Phillips might be 
influencing mathematics motivation through her instruc- 
tional practice. The research team focused observations on 
teacher practice as a means to confirm student perceptions 
of practice in interviews. Three major themes, described 
below, emerged from these observations that served to 
confirm student descriptions of how they experienced 
mathematics in interviews: (a) Respect for peers, (b) 
Increased discourse, and (c) Instruction and management 
decisions. 

Respect for peers. \t was evident that Ms. Phillips had 
prioritized setting mathematical norms with her students 
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such as fostering a sense of respect between herself and 
her students and among the students themselves. Often 
Ms. Phillips reminded students of the need to respect each 
other’s perspectives and ideas during group discussions 
and that if alternative arguments are presented, they must 
be presented with justifications. It was also clear that stu- 
dents in these classes were not afraid of making mistakes 
and saw mistakes as an opportunity to help each other 
rather than judge each other negatively. 

Increased discourse. Students across all observations 
showed the ability and the willingness to engage in dis- 
cussions with each other and with their teacher during 
mathematical tasks. The majority of instructional time was 
spent in whole group or small group discussions where the 
teacher asked open-ended questions eliciting student 
responses. The students, however, also felt comfortable 
presenting alternative arguments to the teacher and chal- 
lenging her ideas as well as those from their peers. While 
teacher-to-student and student-to-student interactions can 
be commonplace in the mathematics classroom, these 
fifth-grade students were especially comfortable engaging 
in student-to-teacher interactions. 

Instruction and management decisions. There was a 
dichotomy between Ms. Phillips’ instruction during the 
introductory phase of the lesson in the first 20 minutes 
after students enter the classroom and during the main 
phase or what she calls her “actual” lesson (the remaining 
40 minutes of instruction). This dichotomy was present for 
every classroom observation. Ms. Phillips essentially 
began every lesson with 20 minutes of test preparation for 
the upcoming state standardized test for fifth grade. 
Instruction during this review period was didactic in 
nature where Ms. Phillips presented problems and asked 
students to solve individually. Then as a class, they 
reviewed the problems with Ms. Phillips providing addi- 
tional information about test-taking skills, such as omit- 
ting improbable responses or checking work repeatedly. 

During the second part of the lesson, Ms. Phillips relied 
on an inquiry-based mathematics curriculum for the 
overall design of the mathematical tasks that were imple- 
mented and the questions that she asked as students 
engaged in the task and while reflecting on the task. A 
review of the curriculum revealed that Ms. Phillips stayed 
fairly close to the original lesson design written in the 
textbook. It is unclear if Ms. Phillips would rely on didac- 
tic instruction, similar to the first fifteen minutes of each 
lesson, if she did not have the support of the textbook to 
guide lesson planning and questioning during lesson 
implementation. It is also unclear if Ms. Phillips’ instruc- 
tion during the first third of each lesson was the result of 
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increased pressure during annual testing time or if it was 
her natural practice. When Ms. Phillips was teaching in a 
student-centered manner, she often encouraged students to 
make connections between what they have experienced in 
school or at home with what they are learning in the 
lesson. Students seemed highly engaged and excited when 
they were able to discuss examples from their own expe- 
riences. In addition to these intentional connections, Ms. 
Phillips demonstrated the ability to make instructional 
decisions based on assessment gathered during lessons. 
Ms. Phillips taught a similar lesson to all of her students 
throughout the day; however, she was able to improve 
these lessons consistently based on her observations of 
students during tasks. Finally, Ms. Phillips displayed a 
positive disposition throughout all observations. She dem- 
onstrated a love for mathematics by showing enthusiasm 
about discussions and mathematical tasks and she 
managed behavior problems (that were typically the result 
of overexcited students getting ready to engage in a task) 
with positive behavior techniques. 

Connections between interviews and _ observa- 
tions. The classroom observations primarily related to 
four themes emerging from student interviews: (a) role of 
the teacher; (b) role of achievement; (c) classroom envi- 
ronment; and (d) connections. Consistencies were 
observed between the paired interviews with fifth-grade 
students and the classroom observations. Under the “role 
of the teacher” theme, students described the desire for 
their teacher to encourage discourse through group work 
and to implement hands-on, challenging tasks, all of 
which were observed during lessons. However, students 
often described their role as being passive in the inter- 
views (where the teacher would tell them step by step what 
to do in their groups). In observations, when Ms. Phillips 
was implementing a lesson rather than a task related to test 
preparation, students were primarily active, determining 
their own strategies for solving problems. In regard to the 
“classroom environment” theme, students expressed 
feeling comfortable and confident during mathematics 
lessons and that there was a positive group dynamic 
among their peers (for example, engaging in healthy com- 
petition or working with each other to learn from mis- 
takes). Lesson observations indicated that Ms. Phillips 
placed a priority on establishing a level of respect in the 
classroom and an environment that encouraged collabora- 
tion. In the “connections” theme, students expressed the 
need for mathematics lessons to build on their current 
experiences and interests. Ms. Phillips often provided 
opportunities for students to make connections between 
content and context. For example, when discussing 
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capacity, she brought in real-life objects for students to use 
in exploration and later asked them to bring in objects 
from home. The primary difference that occurred between 
the lesson observations and the paired interviews was Ms. 
Phillips’ view of assessment and the role it plays in deter- 
mining student success. Students often described Ms. Phil- 
lips as placing a low priority on testing, as stated in the 
“role of achievement” theme, but almost a third of every 
observation was spent preparing students for the state stan- 
dardized test. This disconnection could indicate that Ms. 
Phillips altered her instructional practice because of the 
time of year the observations occurred (in the month 
leading up to the implementation of the state standardized 
test). Regardless of this difference, findings determined 
through classroom observations seemed to correspond 
with four of the overarching themes developed as a result 
of the paired interviews that related to classroom practice. 


Discussion and Implications for Practice 

The qualitative analysis of student interviews revealed 
several key themes, which were influential in these stu- 
dents’ high reported levels of motivation in mathematics. 
While these findings are specific to a particular grade level 
of students in one setting, they could inform the identifi- 
cation of factors that sustain student motivation in math- 
ematics during the progression from early to late 
elementary grades. In turn, these findings could also 
provide insight into factors that could contribute to sup- 
porting student motivation for mathematics across the 
transition to middle grades. 

Central themes across participants included opportuni- 
ties for self-direction and peer collaboration during 
instruction, the critical link between current mathematical 
topics and future applications, and the importance of posi- 
tive perceived teacher interpersonal interactions. In addi- 
tion, students placed a particular priority on the need to 
deemphasize the importance of standardized testing. 
While students valued high grades in mathematics as a 
measure of success (and therefore, higher levels of moti- 
vation), they were motivated by the fact that Ms. Phillips 
did not emphasize test grades, but rather performance 
during tasks as a measure of success. This emphasis on 
personal performance may have fostered a mastery goal 
orientation in these fifth-grade mathematics students. 
Research has shown that a mastery orientation to learning 
can support students’ motivation, particularly their effi- 
cacy and value for a content area (Midgley, Kaplan, & 
Middleton, 2001). Students focused on test performance, 
specifically in relation to competition with other students, 
tend to report lower levels of academic motivation and 
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display lower levels of efficacy for their academic compe- 
tency (Midgley et al., 2001). 

These findings point toward the important part that 
teachers play in engaging students in active roles during 
mathematics instruction, which is consistent with other 
studies of mathematics motivation (Byrnes & Wasik, 
2009). However, while many studies point to teacher 
beliefs and instructional practices as influential toward 
motivation, this study attempts to disentangle how teach- 
ers can best support student motivation through instruc- 
tional practices. Teachers play a central role in helping 
develop effective problem-solving strategies, which in 
turn provide students with the tools they need to operate 
more independently during instruction. Instruction in 
effective problem-solving strategies may also serve to 
elevate students’ self-efficacy in mathematics as they 
develop skills for success as independent learners. In addi- 
tion, teachers can provide students with a feeling of inde- 
pendence and self-direction by providing open-ended 
assignments and choice among several viable options for 
extending instruction on topics in mathematics. They can 
also build environments that encourage substantive dis- 
course and that focus on engaging student interest by 
making connections during lessons. 

Student interviews and classroom observations suggest 
that Ms. Phillips contributed to the overall high levels of 
mathematics motivation for fifth-grade students in this 
setting. However, while teacher practice influenced student 
motivation, it was not the only factor present. Student 
interviews also indicated that students’ present and future 
value of mathematics, along with their self-efficacy for 
mathematics content, also contributed to their levels of . 
motivation. In terms of present value of mathematics, these 
students identified being presented with a challenge, 
finding different ways to solve problems, and high levels of 
persistence as key characteristics to enjoying math. In 
particular, when students discussed whether or not they felt 
like they were good at math, persistence and feeling like 
they could succeed at solving problems were key indicators 
of self-efficacy toward mathematics content. In addition, 
students saw mathematics as something that was important 
to learn now because it will be useful in the future. These 
students saw themselves entering into careers where math- 
ematics might be helpful in succeeding. These findings 
indicate that students in this setting had high levels of 
confidence in their abilities to succeed. Due to the nature of 
the research questions focusing on highly motivated stu- 
dents, it is unclear if students with lower motivation would 
have the same levels of confidence, however, past research 
would indicate otherwise (Pintrich, 2003). 
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Figure J. Potential model for building elementary mathematics motivation. 


The themes emerging from the paired interviews and 
triangulated with the classroom observations form the 
basis for the potential motivation model displayed in 
Figure 1. This model incorporates three main components 
that could potentially build motivation in elementary age 
students: (a) Establishing a safe mathematical environ- 
ment; (b) Encouraging students’ value for mathematics; 
and (c) Reducing external stress related to mathematics. 
This model is meant to inform teachers about ways they 
could potentially increase levels of students’ mathematics 
motivation in their own settings. The three components of 
the motivation model are intertwined because they all 
revolve around teacher practice. In fact, components one 
and three (“Establishing a safe mathematical environ- 
ment” and “Reducing external stress related to mathemat- 
ics”) were directly identified in classroom observations 
during the course of this study. In terms of establishing a 
safe mathematical environment, students identified Ms. 
Phillips’ ability to make connections about mathematical 
content and real-world ideas that were of interest to them. 
In addition, students discussed how Ms. Phillips always 
allowed them to work through tasks together and that the 
classroom was an open space for discussion. Finally, stu- 
dents identified Ms. Phillips’ ability to provide consistent 
encouragement and promote confidence in students during 
mathematics lessons. In terms of reducing external stress 
related to mathematics, students described how Ms. Phil- 
lips would allow them to redo problems on tests or take 
tests home to encourage success. They also identified Ms. 
Phillips’ ability to increase their comfortable level during 
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lessons by making mistakes an opportunity for learning. 
While “Encouraging student’s value for mathematics” was 
not specifically observed in practice, it was clear from 
student interviews that teachers should both display and 
encourage their students to develop present and future 
value for mathematics. Students described Ms. Phillips as 
someone who loved math and who promoted the utility of 
mathematics in their own lives. 


Limitations and Recommendations for 
Future Research 

These results provide insight in regard to domain- 
specific motivation in mathematics. However, this study 
examined highly motivated mathematics students in one 
setting at one point in time. Further, these students all had 
the same mathematics teacher when this study occurred. 
There were no students with low motivation levels 
sampled for analysis in this study. It is unclear why there 
were students in this grade with lower levels of motivation 
(14% across the fifth grade) because the research ques- 
tions did not call for inclusion of students with low levels 
of motivation. However, possibilities for low mathematics 
motivation based on previous research could include 
factors external to the classroom or previous classroom 
experiences. Future studies should compare low and high 
levels of motivation to determine if the proposed motiva- 
tion model would apply for all students. The use of the 
EMMI provided the research team with information about 
the fifth grade students in this setting that they may not 
have found by conducting a phenomenological exploration 
in isolation. Future studies could utilize the EMMI as a 
method of exploring strategies for participant selection, as 
was done in this study. In addition, this study only focused 
on motivational factors and did not seek to make any 
connections between levels of mathematics motivation and 
student achievement. Future research is necessary to deter- 
mine the correlation between these two factors and, in 
particular, how student achievement and levels of motiva- 
tion change based on teacher practice. Finally, the motiva- 
tion model described above is developed based on the 
perceptions of students in this particular setting. Future 
research is necessary to explore the generalizability of this 
potential model. 
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The primary purpose of this study was to examine the ways in which a 12-week afterschool science and engineering 
program affected middle school students’ motivation to engage in science and engineering activities. We used current 
motivation research and theory as a conceptual framework to assess 14 students’ motivation through questionnaires, 
structured interviews, and observations. Students reported that during the activities they perceived that they were 
empowered to make choices in how to complete things, the activities were useful to them, they could succeed in the 
activities, they enjoyed and were interested in the hands-on activities and some presentations, they felt cared for by the 
facilitators and received help when they were stuck or confused, and they put forth effort. Based on our examination of 
data across our three data sources, we identified motivating opportunities that were provided to students during the 
activities. These motivating opportunities can serve as examples to help both formal and informal science educators 
better connect motivation theory to practice so that they can create motivating opportunities for students. Furthermore, 
this study provides a methodological example of how students’ motivation can be examined during the context of 


authentic science and engineering instruction. 


More scientists and engineers are needed in the United 
States to fill the current demand (President’s Council of 
Advisors on Science and Technology, 2010). Because 
researchers have identified middle school as a critical time 
to interest students in science, technology, engineering, 
and mathematics,;(STEM) topics and careers (e.g., Tai, 
Liu, Maltese, & Fan, 2006), we implemented an 
afterschool curriculum titled Studio STEM to interest and 
motivate students to engage in STEM-related fields and to 
teach basic science concepts. We hypothesized that the 
synergy of the curriculum, activities, technology, and 
mentoring in an informal setting could motivate students 
from rural, economically depressed communities. 

The primary purpose of this study was to examine the 
ways in which a 12-week implementation of Studio STEM 
affected middle school students’ motivation and engage- 
ment in science and engineering activities. We used 
current motivation research and theory as a framework to 
assess students’ motivation. We surveyed, interviewed, and 
observed students participating in Studio STEM to assess 
factors that influenced different aspects of their motiva- 
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tion. Our analysis revealed examples of how informal 
science instruction can motivate middle school students, 
and we believe that understanding these motivating oppor- 
tunities can help educators and researchers better connect 
motivation theory and science instruction. 


Background 

Studio STEM 

Studio STEM was a 3-year afterschool and summer 
program that used a design-based approach and an inter- 
disciplinary curriculum to help middle school students 
learn about energy conservation. The principal goal of 
Studio STEM was to engage students in interesting proj- 
ects related to environmental issues that allowed them to 
acquire critical knowledge, skills, and dispositions. Using 
a social constructivist approach (Vygotsky, 1978), teach- 
ers (referred to as site leaders) and engineering and 
science undergraduates (referred to as facilitators) worked 
with upper-elementary and middle school students on 
a curriculum with authentic problems in energy 
sustainability. Studio STEM was based upon the following 
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design principles: (a) an interdisciplinary curriculum to 
interest and engage students in fundamental concepts in 
alternative energy resources, energy transfer, and energy 
sustainability through problems related to environmental 
issues that affect animals; (b) an active, inquiry-based 
learning experience through engineering design activities 
to engage students in real-world projects that promote 
conceptual change and problem-solving strategies; (c) an 
informal learning arrangement organized to approximate a 
design studio to allow creative exploration and exchange 
of problem strategies and solutions; (d) appropriation of 
Internet resources and social network forums to expand 
the learning space; and (e) a workforce development- 
focused endeavor to serve rural Appalachian communities 
by training middle school teachers and undergraduate 
facilitators, as well as cooperating with industry, institu- 
tional, and community partners. 

Learning Through Design-Based Science and 
Engineering Activities 

Studio STEM employed active, inquiry-based learning 
through science and engineering design activities that 
engaged students in authentic problems to promote moti- 
vation, conceptual change, and design thinking. The learn- 
ing strategies that guided this project reflect the vision 
detailed in the National Academy of Engineering and 
National Research Council’s (NRC) publication, Engi- 
neering in K-12 Education (Katehi, Pearson, & Feder, 
2009), and the Next Generation Science Standards (NRC, 
2013). These calls for reform emphasize student-centered 
environments where active inquiry is a primary vehicle for 
learning (American Association for the Advancement of 
Science, 1993; NRC, 2013). 

Design-based learning is a form of problem-based 
learning. When students engage in design-based learning, 
relevant problems provide the context and motivation for 
all the activities that follow (Barrows & Tamblyn, 1980). 
Students learn skills and facts as they progress through the 
process of solving the problem with a design approach. In 
a sense, Studio STEM provided a “practice field” (Barab 
& Duffy, 2000) where participants engaged in activities 
that simulated ones that can be found in the real world. 
These practice fields emphasized (a) ownership of the 
inquiry process, (b) an opportunity for reflection, (c) 
coaching and modeling of thinking skills, (d) collaborative 
and social activities, (e) learning that is motivating, and (f) 
problems that are not oversimplified (Barab & Duffy, 
2000). Moreover, the use of authentic problems allowed 
participants to apply design thinking to their lived experi- 
ence and, thus, better understand the usefulness of what 
they were learning. Researchers (e.g., Fusco & Barton, 
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2001) have noted that students’ perceptions of the useful- 
ness of the problem-solving process were centrally con- 
nected to the ways that they began to see a future in STEM 
careers. For example, Basu and Barton (2007) argued that 
students from economically depressed communities 
develop a sustained interest in science when learning 
experiences are connected with their own futures and 
when they can envision their role in designing solutions to 
authentic problems. 


Theoretical Framework 
The MUSIC Model of Motivation 

The MUSIC™ Model of Motivation (Jones, 2009, 
2010b) was designed to help instructors apply current 
motivation theories by designing instruction that engages 
students in learning. The MUSIC model consists of five 
components that have been derived from research and 
theory as ones that are critical to student engagement in 
academic settings: eMpowerment, Usefulness, Success, 
Interest, and Caring (MUSIC is an acronym). Validity 
evidence indicates that these five components are corre- 
lated, yet distinct (Jones & Skaggs, in press; Jones & 
Wilkins, 2013a, 2013b). Five key principles of the MUSIC 
model are that students are more motivated when they 
perceive that (a) they are empowered, (b) the content is 
useful, (c) they can be successful, (d) they are interested, 
and (e) they feel cared for by others in the learning envi- 
ronment. The MUSIC model does not introduce new moti- 
vation constructs; rather, it presents instructors with 
guidance in designing instruction (e.g., Jones, 2009, 
2010b) by summarizing decades of research in motivation 
and translating it to language that is more understandable 
to instructors than the jargon used by motivation research- 
ers (e.g., self-efficacy, attainment value). When instruc- 
tional environments are designed using the MUSIC model, 
research and theory predict that students will be more 
engaged. The MUSIC model can also serve as a tool for 
researchers to use in understanding the impact of instruc- 
tion on students’ motivation (e.g., Jones, 2010a; Jones, 
Epler, Mokri, Bryant, & Paretti, 2013; Jones, Osborne, 
Paretti, & Matusovich, 2014). 

The empowerment component of the MUSIC model 
refers to the amount of perceived control that students 
have over their interactions with their learning environ- 
ment (Jones, 2009). Research related to empowerment 
has been conducted by researchers studying autonomy 
within the framework of self-determination theory (Deci 
& Ryan, 2000). Instructors can empower students by 
supporting their autonomy, such as by providing them 
with choices and the ability to make decisions. The 
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usefulness component of the MUSIC model involves the 
extent to which students believe that the coursework 
(e.g., assignments, activities, readings) is useful for their 
short- or long-term goals. Students’ goals are important 
because their motivation is affected by their perceptions 
of the usefulness of what they are learning for their 
future and in the real world (Wigfield & Eccles, 2000). 
One implication is that instructors need to ensure that 
students understand the connection between the 
classwork/activities and their goals. The success compo- 
nent of the MUSIC model is based on the idea that stu- 
dents need to perceive that they can succeed at a task if 
they put forth the appropriate effort. Self-perceptions of 
competence (i.e., one’s beliefs about one’s ability to 
succeed) are central to many motivation theories, such as 
self-concept theory (Marsh, 1990), self-efficacy theory 
(Bandura, 1986), self-worth theory (Covington, 1992), 
goal orientation theory (Ames, 1992), and expectancy- 
value theory (Wigfield & Eccles, 2000). Instructors can 
foster students’ success beliefs in a variety of ways, 
including making the course expectations clear, challeng- 
ing students at an appropriate level, and providing stu- 
dents with feedback regularly. The interest MUSIC 
component includes situational interest, which refers to 
the immediate, short-term enjoyment of instructional 
activities. Interest is related to many positive outcomes, 
including attention, memory, comprehension, deeper 
cognitive engagement, goal setting, learning strategies, 
and achievement (Hidi & Renninger, 2006; Schunk, 
Meece, & Pintrich, 2014). Instructors can trigger stu- 
dents’ situational interest by designing instruction and 
class work that incorporates elements such as novelty, 
social interaction, games, humor, or surprising informa- 
tion, or that engenders emotions (Bergin, 1999). The 
caring MUSIC component is derived from research in 
the areas of belongingness, relatedness, connectedness, 
affiliation, involvement, attachment, commitment, 
bonding, and sense of community (e.g., Baumeister & 
Leary, 1995; Deci & Ryan, 2000). Caring can be divided 
into two components: academic caring and personal 
caring (Jones & Wilkins, 2013a, 2013b). When students 
have close relationships with their teachers and peers, 
they tend to be more engaged in the classroom and 
perform better academically (Bergin & Bergin, 2009). 
Academic caring specifies that instructors need to dem- 
onstrate to students that they care about whether or not 
their students successfully meet the course objectives. 
Personal caring includes the idea that students need to 
perceive that their instructor and/or other students care 
about their welfare. 
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Research Question 

Our research question for this investigation was: In what 
ways does Studio STEM affect middle school students’ 
motivation and engagement in science and engineering 
activities? To answer this question, we studied a 12-week 
implementation of Studio STEM by analyzing data col- 
lected through questionnaires, structured interviews, and 
observations. 

Method 
Participants ‘ 

We obtained a convenience sample from students 
enrolled in a K-7 school in rural, southwest Virginia that 
included all 14 (4 girls and 10 boys) of the sixth- and 
seventh-grade students who participated in Studio STEM 
at this school. Students met once a week after school seven 
times in the fall and five times in the spring (each session 
lasted 90 minutes). 

Save the Seabirds Curriculum 

In our study, we examined the 12-session Save the Sea- 
birds curriculum module (available at http://www.auburn 
.edu/~cgs0013/engineering.htm) in which students desig- 
ned, built, tested, and re-tested a mini solar-powered car to 
pull as much weight as possible. They also tested different 
solar panels, motors, and methods for transferring motion 
from motor to wheels. Activities targeted alternative con- 
ceptions of force and motion, and students learned about 
petroleum use and the Law of Conservation of Energy in the 
context of a scenario in which solar power could be used to 
reduce dependence on petroleum, which can spill and harm 
seabirds. The instructors included the Studio STEM project 
leader, a science teacher from the school where Studio 
STEM was implemented, and two undergraduate engineer- 
ing students from a nearby university. 

The format of the sessions varied. In some sessions, the 
teacher presented slides on a projector to orient students to 
the topics and to provide content information; afterwards, 
students engaged in activities. In other sessions, especially 
in the last five sessions, the teacher conducted fewer pre- 
sentations and students spent more time engaged in the 
activities. 

Data Sources and Collection 

We collected and analyzed both qualitative and quanti- 
tative data, and then triangulated those data to form a 
deeper understanding of students’ motivation and engage- 
ment in Studio STEM. Although our sample size was 
limited, we posit that it was sufficient considering our 
focus on qualitative methods (Glaser & Strauss, 1967). 

Questionnaire. All 11 students who attended the last 
Save the Seabirds session completed a questionnaire that 
included items about their perceptions of the 12 sessions. 
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Students reported their perceptions of each of the MUSIC 
model components using the MUSIC Model of Academic 
Motivation Inventory—Middle School version (Jones, 
2012), which includes four empowerment items, three use- 
fulness items, four success items, three interest items, and 
four caring items. This inventory has been shown to 
adequately represent the five-factor structure of the 
MUSIC model (i.e., empowerment, usefulness, success, 
interest, and caring) with students in the fifth, sixth, and 
seventh grades (Jones & Wilkins, 2013b, reported empow- 
erment & = .70, .72; usefulness & = .82, .80; success O = 
.79, .84; interest & = .78, .77; and caring a = .78, .85; see 
the Appendix for sample items). The questionnaire also 
included a 4-item scale that measured the amount of per- 
ceived effort that students put forth in Studio STEM. This 
measure is based on the 5-item Effort/Importance scale 
that is part of the Intrinsic Motivation Inventory (Plant & 
Ryan, 1985) and has been shown to have adequate psy- 
chometric properties (Jones, 2010a). All of the items on 
the questionnaire referred to the “Studio STEM activities” 
related to the Save the Seabirds curriculum and were rated 
on a 6-point Likert-type scale (1 = strongly disagree, 2 = 
disagree, 3 = mostly disagree, 4 = mostly agree, 5 = agree, 
6 = strongly agree). 

Interviews. Three rounds of structured interviews were 
conducted with students. The first round of interviews 
occurred during the first session (n = 14; 100% of the 
students enrolled in the program), the second round of 
interviews occurred during the seventh session (n = 11; 
79% of the students), and the third round of interviews 
occurred after the twelfth session when the program was 
complete (n = 9; 64% of the students). We interviewed all 
of the students who attended the sessions in which we 
conducted the interviews; some students were absent 
during the second and third interviews. Each interview 
lasted from 10 to 20 minutes. The interview protocol was 
structured, with two to five questions pertaining to each of 
the five MUSIC components. The goal of the interviews 
was to uncover students’ perceptions of the presence or 
absence of the five MUSIC components in the curriculum 
activities. 

The purpose of the first interview was to assess why 
students decided to participate in the program, what they 
knew about the program, their beliefs about the value of 
science, and their science ability beliefs. The second and 
third interviews began by showing students a summary list 
of what they had done during the sessions since the last 
time they were interviewed. They were asked to think 
about the sessions in which they participated during that 
time period only, and to respond to the questions in refer- 
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ence to those sessions only. Prior to each open-ended 
question, the interviewer showed students blocks consist- 
ing of one, two, or three connected, plastic cubes labeled 
“Not very” (or “Very little”), “Somewhat” (or “Some’’), or 
“Very” (or “A lof’), respectively. An example question for 
the interest component was: 


The small block means that the presentations were not 
very interesting to you, the medium block means that 
the presentations were somewhat interesting to you, 
and the large block means that the presentations were 
very interesting to you. Which block do you think 
shows how interesting the presentations were to you? 
[Response] Okay, you picked the [small, medium, 
large] block. What was interesting about the 
presentations? 


This procedure of showing the blocks, explaining what 
each meant, asking students to choose one, and then 
asking students to explain their response was used for each 
MUSIC component. 

Observation data. We collected observation data 
during all sessions. All observers were trained on data 
collection procedures, including the use of two forms 
(described next). The observers were selected because 
they had received extensive training pertaining to the 
MUSIC model and had studied academic motivation at the 
doctoral level. The observers did not interact directly with 
the students or instructors. 

MUSIC Model Observation Form. One researcher 
completed a MUSIC Model Observation Form during 
each session to assess the prevalence of the five MUSIC 
model motivation components during the session. The 
form allowed the observer to document (a) the time that 
the evidence was observed, (b) indicators of the MUSIC 
components, (c) a description of the activity in which 
the indicator occurred, (d) relevant quotations, and (e) the 
names of the students involved. The indicators allowed the 
observer to document when he/she observed evidence that 
students perceived that (a) they were empowered, (b) the 
content was useful, (c) they could be successful, (d) they 
were interested, and/or (e) they felt cared for by others in 
the learning environment. Because these MUSIC compo- 
nents have been shown to lead to motivation and engage- 
ment (Jones, 2009), the documentation of these MUSIC 
indicators provided some evidence of student motivation 
and engagement. Each observer was trained to look for 
student-student and _  instructor—student interactions, 
instructional practices, and student behaviors that could 
suggest that elements of the instructional environment 
facilitated their motivation. For example, observers 
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attended to students’ verbalizations and/or behaviors that 
suggested that they felt success (e.g., a student pumping 
fists and saying “Yes!” after succeeding at a task). As 
another example, an observer could indicate that empow- 
erment was supported if he/she noted that the instructors 
provided students with a meaningful choice in how to 
design their solar-powered cars. 

General Observation Form. One researcher completed 
the General Observation Form during every session to 
collect information related to the session’s presentations 
and activities, as well as students’ motivation and engage- 
ment. Observers were trained to use the form, which was 
arranged into three sections: (a) Context, which allowed 
for the collection of data relevant to the physical layout of 
science activity spaces and the circumstances that 
impacted these activities; (b) Description of Activities, 
which allowed for a description of the students’ activities; 
and (c) Behavioral Engagement, which allowed for the 
collection of curricular and/or instructional elements that 
led students to be engaged behaviorally. 

Data Analysis 

Descriptive statistics were computed for the question- 
naire scales. The interviews were audio-recorded and tran- 
scribed verbatim. Two of the authors developed a list of 
codes using thematic whole text analysis, which was 
informed by the analytic procedure developed by Glaser 
and Strauss (1967). Next, two authors coded all interviews 
according to the identified codes and compared results. 
When there was a difference in the coding, the items were 
discussed and a final code assigned. We calculated the 
inter-rater reliability as 86.3% for the first interview, 82.5% 
for the second interview, and 85.7% for the third interview. 

Trustworthiness and credibility were enhanced, and 
results deepened, through triangulation with the results of 
the observation ferms. We coded all motivation-related 
observations noted in both forms. For our analysis, we 
used the existing codes developed from the student inter- 
views, which we coded prior to coding the observations. 
We matched the relevant excerpts within the observation 
forms to the interview codes and created new codes when 
we did not have an interview code that matched the obser- 
vation excerpts. When excerpts on the observation forms 
from the same session were labeled with the same code, 
we “counted” the code only once (1.e., the same code was 
not counted more than once on any one session). 


Results 
Questionnaire 
On the 6-point Likert-type scale, students (NV = 11) gen- 
erally reported that, during the Studio STEM (Save the 
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Seabirds) activities, they perceived that: they were 
empowered (M = 5.23, SD = 0.74), the activities were 
useful to them (M = 4.67, SD = 0.97), they could succeed 
in the activities (VM = 5.55, SD = 0.62), they enjoyed and 
were interested in the activities (M= 5.42, SD = 0.60), they 
were cared for by the facilitators (M = 5.66, SD = 0.57), 
and they put forth effort (WV = 5.48, SD = 0.68). On 
average, students reported scores greater than 5 (agree) for 
their effort and for all of the MUSIC component subscales, 
except for usefulness. 

Interviews and Observations 

During the first interview, all of the students reported 
that they had participated in Studio STEM the prior year. 
When asked with an open-ended question why they 
enrolled again in the current year, many of the students 
(71%) said that it was because they had enjoyed partici- 
pating in the program the prior year. We also asked them: 
“Is there something about [Studio STEM] that interests 
you?” Students reported that they were interested in solar 
cars and designing and building things, and that they liked 
seabirds and teamwork. 

Results of the second interview (after 7 weeks of the 
program) and third interview (after 12 weeks) are sum- 
marized in Tables1 and 2. Students’ responses vary 
somewhat for the two interviews because the presenta- 
tions and activities differed throughout the 12 weeks. The 
results of the observations are also provided in Table 2. 
In all, we created 118 codes, with 102 codes -for the 
interviews and 16 additional codes for the observations 
that were not present in the interview codes. Of the 118 
codes, 54 codes (45.8%) were used for both the inter- 
views and observations. Of the 102 codes related to only 
the interviews, we observed 54 (52.9%) and did not 
observe 48 of those codes (40.7%) because they were 
yes/no responses or questions pertaining to the future 
(thus, they were impossible to observe). Of the 33 
codes that could be observed, 26 (78.8%) were men- 
tioned by only one student during one of the interviews 
(or 54.2% out of the 48 codes), five were reported by two 
students, one was reported by three students, and one 
was reported by four students. These findings suggest 
that the majority of the codes that were not observed 
were cited by very few students or could not be 
observed, which indicates that those codes were not 
prevalent in the program. Most of the interview 
responses (86.1%) and observations (93.5%) were con- 
sistent with the MUSIC model in that they documented 
instructional practices that can facilitate students’ moti- 
vation (e.g., students perceived that they could be 
successful). 
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Table 1 
Responses by students when shown blocks labeled with descriptors 


Interview 2 


Not Very . Somewhat 

MUSIC component (Very Little) (Some) 
eMpowerment 1 3 
Usefulness Now 2 + 

Future 0 5 
Success 0 2 
Interest Presentations 0 5 

Activities 0 0 
Caring Facilitators 0 0 

Teammates 1 6 
Effort 0 1 


Interview 3 


Very Not Very Very 
(A Lot) (Very Little) Somewhat (A Lot) 
a 0 1 8 
5 4 5 0 
6 0 5 4 
9 0 4 5 
6 — == Es 
11 0 3 6 
11 0 0 9 
4 Ba a wits 
10 0 0 9 


Note. A dash (—) indicates that students were not asked about that component. 


Because research and theory indicate that students put 
forth effort and engage in activities when they perceive 
that the activities have high levels of one or more of the 
MUSIC model components (Jones, 2009), we examined 
each of the MUSIC model components to gain a deeper 
understanding of which elements of the Studio STEM 
activities contributed to students’ motivation. We discuss 
the findings in the following sections in the order of the 
MUSIC acronym for organizational purposes only; we do 
not mean to imply that any one of the components was 
more important than the others in motivating students. 

Empowerment. Related to empowerment, we asked 
students about the choices they had in the Studio STEM 
sessions. Students discussed the broader choices that they 
had, such as choices related to how to build assigned 
projects and the use of materials (e.g., one student 
responded, “Well, I mean they gave you the requirements 
of the activity, but then basically you got to choose how 
you wanted to do it, as long as.it met the requirements”). 
They also discussed specific choices, such as how many 
cubes to put in a cup when building a lift (e.g., another 
student explained, “Well, you got to make choices on how 
many cubes you could put into the basket, you got a choice 
on how to connect your circuit’s wires to make your solar 
panel work”), the choice of their group partners, and what 
to write on their storyboards (i.e., posters filled-in each 
day that tracked their learning). These student responses 
were corroborated with the observation data, as shown in 
Table 2. The observers also documented that students were 
empowered in several additional ways, such as in selecting 
their seating arrangements and controlling the pace and 
direction of the lessons and activities. One often-cited 
strength of design-based science is that it is “student- 
centered” because it allows students to direct their own 
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learning (Kember, 1997). Our data are consistent with this 
claim and provide specific examples as to when students 
had choices. 

Usefulness. Students reported that they found the 
program to be somewhat useful to their lives at that time 
and in the future. When asked about the aspects of the 
program that were useful to their lives currently, different 
students reported different topics that were useful to them, 
resulting in a list of activities, such as learning about 
motors, electricity, energy, gears, tires, and combustion 
engines. Three students also noted that what they learned 
in the program had been useful in their science classes, 
indicating that they found the curriculum concepts to 
transfer to their in-school science learning. A few students 
thought that the program was not very useful to their 
current lives. For example, one student explained that the 
program was not very useful to his current life because he 
could not build anything (e.g., a solar car) outside of 
school: “Because I couldn’t build really anything right 
now, and solar cells are really expensive and I wouldn’t 
really know how, what they’d be used for... . But it’s 
something in the future I’d need to know.” Another student 
noted that the program did not overlap with science class 
topics. In contrast, a student explained that the program 
was not very useful because it was only applicable to 
science class. Interview results are largely consistent with 
the observations, and the observations added that the 
facilitators often provided rationales for rules and related 
the program to the real world, which could add to students’ 
perceived usefulness. 

When asked about how what they did was useful for the 
future, some students reported that it was useful for getting 
into college or succeeding in college, and others noted that 
it was useful for becoming a scientist or engineer (e.g., one 


409 


Elements of Activities That Affect Motivation 


Table 2 
Interview questions and number of interview responses and observations for each code 
Interview After: Observations Duringy: 
First Last First Last 
Questions and Codes for the MUSIC Model Components (Organized by MUSIC Component) 7 weeks 5 weeks 7 weeks 5 weeks 
Empowerment 
What choices did you have in Studio STEM (in these sessions)? 
Choice of how to build the solar cars 0 8 0 3 
Choice of how to complete things 2 > 2 2 
Choice of how to build my motor to pull up the cup with cubes 5 0 2 0 
Control over the use of materials 0 0 4 3 
Choice of how to put the gears together 2 0 1 1 
Choice over researching on electricity, energy, batteries, and solar panels 3 0 0 1 
Choice to write on the storyboards 0 | 1 1 
Choice of group l 0 2 1 
Usefulness (for short-term goals) 
What have you done in Studio STEM (in these sessions) that’s useful to your life right now? 
Learning about motors S 0 1 0 
Learning about electricity 3 0 1 0 
The rationale for the rules or expectations was provided 0 0 1 3 
Learning about solar cells or cars 2 0 1 1 
What I learned in these sessions has been useful in my (science) classes 0 3 0 0 
Success 
What made you feel that you could be successful in Studio STEM (in these sessions)? 
The facilitators were helpful (including feedback) 3 2 5 5 
There were clear instructions or expectations 0 0 3 “ 
I am good at science or math 3 1 0 0 
There was an appropriate level of challenge (the facilitators monitored readiness) 0 0 3 3 
I felt successful in working on the solar cars 0 2 0 4 
I could remember and use what we learned (e.g., reviewed tasks, presentations, activities) 2 1 2 0 
I felt successful in working on motors 2 1 1 0 
Interest (presentations) 
What was interesting about the presentations? 
Learning about oil rigs and their effects on animals and the environment 5 0 1 0 
Watching videos on force and combustion engines or how to turn electrical energy into 4 0 1 1 


mechanical energy (e.g., gears) 
Learning about gears 0 3 0 pl 
Interest (activities) 
What was interesting about the activities? 
The activities were hands-on or allowed us to experiment 
Building the car 
Working with motors to lift up the basket with cubes 
Using the Legos 
Measuring the voltage and amps (multimeters) 
Playing with a solar powered car inside and outside (demonstration) 
Caring (instructors), 
How did they (the instructors) show that they cared about your success in Studio STEM? 
They helped when we were stuck, confused, or missed something 
They asked about how we were progressing on the activities 
They wanted us to be successful 
They used teaching methods that would help us understand the content 
They trusted me to make choices (and have good ideas) 
They helped me work on the project when my partner was absent 
Caring (other students) 
How did they (other students) show that they cared about your success in Studio STEM? 
My partner helped me work on the project 2 0. 4 
They wanted me to be successful 2 0 1 


NWONCH 
SCoON ODO 
RereNNOA 
RSP OWOWN 


mee Nnorn 
ocoocor~ya 
ANAWUUN 
BPWNWUaA 


4 
1 
Note. Codes with fewer than a combination of four interview responses and observations are not shown. We divided the observations into those 


made during the first 7 weeks and the last 5 weeks to allow for easier comparison with the interview results. 
tOnly one observation per week was recorded even when more than one observation was noted on the observation rubrics. 


student responded, “If I get a job in engineering ... you out what they wanted to study in college or their future 
have to understand this type of stuff to go to college, geta career choice. These findings are important because they 
good job, and get good grades—stuff like that”). Students indicate that this program played a role in students’ 
also reported more generally that it was useful for figuring thoughts about college in general, as well as about specific 
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career options. For example, one student explained, “I’m 
planning on being a robotics engineer at [University]. 
Whenever you guys came here, I was talking about robots 
and [facilitator] showed me the [university engineering 
program] webpage. I went home, checked it out, and it 
looked really, really cool.” However, it is difficult to deter- 
mine the extent to which students’ beliefs were affected by 
the curriculum content versus the interactions with the 
facilitators, as the facilitators were undergraduate students 
and the program participants worked with them during 
each session. Perhaps talking with, and getting to know, 
undergraduates was also important to their perceptions of 
what it meant to become a college student and, specifically, 
an engineering student. 

The results from the students’ interviews and our obser- 
vations were aligned fairly closely for all of the MUSIC 
model components except for usefulness. The usefulness 
of the activities was more difficult to observe without 
knowledge of students’ interests and goals outside of the 
specific program activities in which they were participat- 
ing. For example, one student discussed that learning 
about motors was useful to him because of his father’s 
work: “My papa works on motors and fluids and stuff, so 
I’m going to help him do that when I get older so it’d be 
good to get to know all about motors and mechanical 
energy now.” This background knowledge was unknown to 
the observer and, therefore, unless the student had verbal- 
ized this fact during the activity, it was impossible to 
document through observation only. 

Success. In general, students believed that they could 
succeed in the program. Some of the most common 
interview responses were that their success _ beliefs 
derived from the helpful facilitators and site leaders (e.g., 
one student explained, “It was pretty hard but the facili- 
tators, they encouraged me.”). Others attributed their 
success to the fact that they were good at science, math- 
ematics, or designing things (e.g., “Partially because I 
am very good at science .... Math is another strong 
point of mine”). Some students noted specific activities 
in which they felt successful, such as in working on the 
solar cars, motors, solar panels, gears, and measuring 
voltage and amps. The observers noted during seven ses- 
sions that there were clear instructions or expectations, 
which could have allowed students to feel successful. A 
few students explained that they did not feel successful 
during specific events, including working on the solar car 
(because it was challenging), using the motors to lift 
baskets of cubes, the quizzes, learning about voltage, and 
after they missed one or more sessions. However, these 
feelings were not common in the interview data, as only 


School Science and Mathematics 


one student cited each issue. Most of the students 
reported that they felt they could succeed in the program, 
which indicates that the curriculum was designed at an 
appropriate level for these students. 

Interest. The students reported that they were inter- 
ested in the program; however, the activities were more 
likely to stimulate their interest than the presentations. 
They perceived the activities to be interesting because 
they were “hands-on” and allowed them to experiment 
with things such as building the car, working with 
motors, using Legos, measuring the voltage and amps, 
playing with a solar car, playing with gears, and making 
a circuit for the solar cell. For example, when asked what 
was interesting about the activities, one student 
explained: “It was very hands-on. We got to do most the 
things, instead of the teacher showing us what it is and 
stuff, we got to actually experiment by ourselves.” When 
asked directly whether there were times when they felt 
that the activities were boring, none of the students 
reported that the activities were boring. However, the 
observers noted that, during one activity in which the 
students were reviewing and refining their solar cars, 
the students were less engaged, some were off-task, and 
the classroom environment was “slightly chaotic.” In this 
case, the observations provided important details about 
the activity that were not recalled by students during the 
interviews. 

Students’ interview responses were different when they 
were asked about the site leader’s presentations. Presenta- 
tions were short (10 minutes or less) and took place at the 
beginning of a session to introduce new ideas or to review 
information. Presentations included images, text, and 
videos, and were designed to be engaging and interactive. 
Several students reported that the presentations were 
boring, especially watching some of the PowerPoint pre- 
sentations. One student mentioned that it was boring when 
the site leader reviewed things that he already knew. 
Although some reviews may have been uninteresting, they 
may have helped students feel successful. That is, maybe 
these reviews were needed to help develop their percep- 
tions of competence and to be successful in future tasks. 
As another example, two students indicated that presenta- 
tions were boring when they did not get enough informa- 
tion or did not understand the content, suggesting that their 
poor ability beliefs negatively affected their experience of 
interest. One of the two students described what was 
boring: “The parts where they were talking about a whole 
bunch of stuff that I didn’t understand. Like, like when we 
were talking about engines; like some stuff was mentioned 
that I just didn’t get.” These findings demonstrate the 
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complex interplay among MUSIC model constructs, such 
as interest and success. 

Nonetheless, the presentations were found to be inter- 
esting sometimes, such as when they were learning about 
topics such as electricity, voltage, current, solar cells, 
motors and engines, and gears. Some students found it 
interesting to watch the videos on force and combustion 
engines, and how to turn electrical energy into mechanical 
energy. These interview findings were consistent with the 
observation findings and indicate that the activities were 
generally more interesting to students than the presenta- 
tions, although parts of the presentations were interesting, 
especially the videos and when they learned something 
new. 

Caring. For the caring component, we assessed 
whether students felt cared for by the site leaders, facili- 
tators, and their teammates, separately. Students typically 
rated the facilitators and site leaders as highly caring. The 
most common way in which students reported that the 
facilitators and site leaders showed that they cared about 
students’ successes was that they helped the students when 
needed. For example, one student said, “If we didn’t 
know something they would help you and get you caught 
up .... They would kind of let you explore and then, even 
though sometimes it didn’t work, they would .. . help you 
figure it out.” Another student similarly explained, “They 
actually helped, they were not just standing back there, and 
they communicated.” The observation data were consistent 
with this interview finding and added other details, such as 
that the facilitators and site leaders asked about how stu- 
dents were progressing on the tasks. Some students also 
reported that the facilitators and site leaders used teaching 
methods that would help them understand the content and 
helped them work on the project when their partners were 
absent. , 

Table 1 shows that the students felt cared for by their 
teammates, although less so than by the facilitators and 
site leaders. Several students reported that their partners 
helped them work on the project, wanted them to be suc- 
cessful, were interested in the project work, or believed in 
them. Some explained that other students in the program 
also cared about their successes. For example, a student 
said, “Especially my classmates, they’re always saying 
‘you can do this,’ and stuff like that. And the other people 
in Studio STEM, they want to succeed too. So they try to 
get other people to succeed.” However, one student spe- 
cifically noted that his teammate did not care about his 
success and another responded that she did not think her 
peers cared about other teams’ successes. Researchers also 
observed that during one session, because of a competition 
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Table 3 
Studio STEM elements, example motivating opportunities, and key MUSIC 
components affected 


Studio STEM 
Elements 


Key MUSIC Model 


Example Motivating 
Components Affected 


Opportunities 
Topic selection * Opportunities to ¢ Usefulness 
learn about STEM 
topics 
Presentations *° Opportunities to ¢ Interest, usefulness 
interest students in 
the topic and show 
the usefulness of it 
Activity * Opportunities to ¢ Empowerment 
choices choose how to build 
the solar car 


Group ¢ Opportunities to ¢ Success, caring 


interactions interact with other 
students in small 
groups 
Instructor ¢ Opportunities for ¢ Success, caring 
support instructors to help 
students 





between teams involving the successful completion of 
their solar cars, some students appeared unconcerned with 
the other teams’ successes. 


Discussion and Implications 

The primary purpose of this study was to examine the 
ways in which Studio STEM affected middle school stu- 
dents’ motivation and engagement in science and engi- 
neering activities. The high means on the questionnaire 
subscales and the high percentage of comments from the 
interviews and observations that were consistent with the 
MUSIC model (81.4%; see Tables 1 and 2) indicate that, 
overall, students were motivated to put forth effort and 
engage in the Studio STEM activities. These findings are 
also consistent with the students’ self-reported effort. In 
the next two sections, we summarize the primary findings 
and offer implications for similar learning environments. 
Motivating Opportunities 

Based on our examination of data across our three data 
sources, we identified motivating opportunities that were 
provided to students in Studio STEM (see Table 3 for a 
summary and some examples). We found that these moti- 
vating opportunities triggered one or more of the MUSIC 
model components and, thus, were opportunities to moti- 
vate or demotivate students. Our data suggest that several 
elements of Studio STEM (more generally) and the Save 
the Seabirds curriculum (more specifically) were particu- 
larly motivating, which we describe with more detail in 
this section. We propose that the following examples offer 
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implications that can be modified by instructors in other 
student-centered and inquiry-based learning environments 
when intentionally designing instruction to motivate and 
engage science students. 

Three primary strengths of the Studio STEM program 
were evident and are applicable to a variety of learning 
environments. The most frequently documented findings 
in both the interviews and observation data were (a) that 
the students perceived the facilitators and site leaders to be 
helpful, including their provision of informational and 
constructive feedback (e.g., facilitators helping students 
when they felt stuck, confused, or when something impor- 
tant was missed); (b) that the activities were hands-on such 
that the students could experiment on their own; and (c) 
that the students had choices in how to complete things. 
Although these three findings were the most prominent in 
both the interview and observation codes, other findings 
were also common in either the interviews or observation 
codes. These findings were organized in two. ways: 
(a) prevalent codes in the observation data pertaining 
to general program features and (b) codes frequently 
noted in the interviews concerning specific activities or 
presentations. 

Several codes related to overall program components 
were particularly prevalent in the observation data because 
they were integrated consistently into the Studio STEM 
program structure; thus, they were observed during mul- 
tiple sessions. Of these, the majority pertains to the caring 
component of the MUSIC model, including the helpful- 
ness and availability of the program facilitators and site 
leaders, who, in addition to helping students, clearly com- 
municated a desire for students to be successful, trusted in 
the students’ ideas and choices, and monitored the stu- 
dents’ progress. In addition, many students highlighted the 
importance of their partners’ support. Each of these speaks 
to a safe learning environment designed to foster positive 
relationships among the students, facilitators, and site 
leaders. Two additional codes were reported frequently and 
were only evident in the observation data: (a) the students 
frequently had control over a variety of materials and 
material choices during multiple activities, which could 
have contributed to their perceived empowerment; and (b) 
the facilitators and site leaders provided clear instructions 
and expectations, which likely supported the students’ per- 
ceived success. These two components are consistent with 
previous research, which indicates that it is important to 
support students’ success while encouraging their 
autonomy (Patall, Sylvester, & Han, 2014), especially 
when effectively implementing inquiry-based learning 
(Tai, Sadler, & Maltese, 2007). 


School Science and Mathematics 


Two activities and one lesson topic were prominent in 
the interviews. The majority of these codes relate to the 
interest component of the MUSIC model. The activities 
involved students directing the design and construction of 
two separate systems. One system was the solar car, which 
was the culminating project in the program in which stu- 
dents were provided a variety of materials (e.g., motors, 
solar panels, Legos) and options for design and construc- 
tion, and worked on their cars during multiple sessions. 
Responses indicate that students’ motivation was sup- 
ported by the activity in general, the choices inherent in 
how to build the cars, and the outside support from facili- 
tators, site leaders, and teammates. The second task 
involved teams designing and assembling a motor- 
powered system for raising a basket full of cubes using 
multiple materials (e.g., gears, Legos, solar panel). The 
level of autonomy built in to this activity was also high- 
lighted in the findings. These two activities were similar in 
several respects: (a) students led the design and implemen- 
tation of the systems following presentations of relevant 
background information in a cumulative manner over mul- 
tiple sessions; (b) the provided materials offered many 
choices such that students not only had control over design 
but also which parts were used or abandoned (e.g., two 
options were provided for both the motors and solar 
panels); and (c) facilitators and site leaders were available 
to support the students’ work and review the content mate- 
rial. Thus, although the two activities may not be appli- 
cable to other learning environments, these more general 
characteristics that we identified could be transferred to 
other environments. In addition to these two activities, 
multiple students agreed that learning about oil rigs and 
the environmental implications of oil spills throughout the 
program was particularly interesting. This suggests that 
connecting lessons and activities to real-world problems 
can positively affect motivation through the interest and 
usefulness components of the MUSIC model. 

Missed Opportunities: Areas for Improvement 

Codes relating to the usefulness component of the 
MUSIC model were the least common of the highest fre- 
quency codes. Thus, it is possible that the program could 
be improved by more often connecting the activities and 
presentations to students’ goals (present and future), their 
lives at home, and the real world. However, it is possible 
that making only a few connections is necessary and that 
more connections would be redundant; this is a point that 
could be examined in future research. Another weakness 
predominant in the interview responses was that multiple 
students reported that their interest suffered during the 
direct instruction aspects of the program. Although 
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presentations may have been necessary for learning, our 
findings suggest that teacher-centered, lecture-based 
methods may not nurture interest as readily as other 
aspects of the program. Given that previous research sug- 
gests that direct instruction and prior knowledge are often 
essential to student success in the ill-structured environ- 
ments typical of problem-based learning (e.g., Kirschner, 
Sweller, & Clark, 2006), we posit that instructors could 
attend to other components of the MUSIC model during 
more direct instruction (e.g., provide rationales to make it 
useful, incorporate meaningful choices) when lessons are 
less likely to trigger interest. 


Conclusion 

The results from each of the data sources were fairly 
consistent and demonstrated that Studio STEM motivated 
middle school students to engage in the science and engi- 
neering activities. However, we also documented some 
ways in which Studio STEM could be improved. Our 
findings indicate that it is possible to motivate students to 
engage with science and engineering concepts in a volun- 
tary, afterschool program. Understanding the motivating 
opportunities we identified can help both formal and infor- 
mal science educators better connect motivation theory to 
practice so that they can create motivating opportunities 
for students. Furthermore, this study provides a method- 
ological example of how students’ motivation can be 
examined during the context of authentic science and engi- 
neering instruction. 
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Appendix 

Example Items from the MUSIC Model of Academic 
Motivation Inventory—Middle School Version (Jones, 
2012) 

Note that these sample items were arranged randomly 
and students rated them on the following scale (see Jones, 
2012, for the full version of the inventory): 





1 Z 3 4 5 6 
Strongly Disagree Mostly Mostly Agree Strongly 
Disagree Disagree Agree Agree 
Empowerment 


I had flexibility in what I was allowed to do during the 
Studio STEM activities. 

Usefulness 

The knowledge I gained in the Studio STEM activities is 


important for my future. 


School Science and Mathematics 


Success 

I was capable of doing well at the Studio STEM activities. 
Interest 

The Studio STEM activities were interesting to me. 
Caring 

The [university] facilitator cared about how well I did in 
the Studio STEM activities. 
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Next Generation Science Standards (NGSS) science and engineering practices are ways of eliciting the reasoning and 
applying foundational ideas in science. As research has revealed barriers to states and schools adopting the NGSS, this 
mixed-methods study attempts to identify characteristics of professional development (PD) that will support NGSS 
adoption and to improve teacher readiness. In-service science teachers from across the nation were targeted for the 
survey and responses represented 38 states. Research questions included: How motivated and prepared are in-service 
7-12 teachers to use NGSS science and engineering practices? What is the profile of 7-12 in-service teachers who are 
motivated and feel prepared to use NGSS science and engineering practices? The study revealed that teachers identified 
engineering most frequently as a PD need to improve their NGSS readiness. High school teachers rated themselves as 
more prepared than middle school and all teachers who use Modeling Instruction expressed higher NGSS readiness. 
These findings and their specificity contribute to current knowledge, and can be utilized by districts in selecting PD to 


support teachers in preparing to implement the NGSS successfully. 


On April 9, 2013, the final draft of the Next Generation 
Science Standards (NGSS) was published. This document, 
the result of collaboration between Achieve, Inc. and 26 
lead states, seeks to provide students nationwide with an 
internationally benchmarked education by articulating 
crosscutting and deeply conceptual science performance 
expectations (NGSS, 2013). Despite the final publication 
of NGSS and the commitment of lead states to adopt the 
standards, little exists in firm implementation strategies 
and timelines. Research has already identified barriers to 
NGSS implementation such as the limited instructional 
time currently devoted to science, the lack of physical 
resources for effective science education, and insufficient 
preparation of elementary teachers (Trygstad, Smith, 
Banilower, & Nelson, 2013). However, teacher prepared- 
ness is a concern at all levels as some current models of 
teaching (fact and lecture based) are incapable of satisfac- 
torily addressing the mandates of NGSS, which emphasize 
conceptual and exploratory learning (Cooper, 2013). Thus, 
there is a national urgency to identify the kind of profes- 
sional development (PD) that will best prepare teachers to 
meet the challenges of the NGSS. It may require a con- 
siderable investment of resources to develop the appropri- 
ate materials and tools to support both teachers and 
students (Wilson, 2013). We must realign the resources 
spent on PD with the demands teachers will face in an 
NGSS classroom, and also conduct the research required 
to learn from it. 

The present study sought to meet the need for research 
on readiness and motivation of middle and high school 
in-service teachers to apply NGSS science and engineer- 
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ing practices in their classrooms. We also attempt to iden- 
tify the characteristics of a teacher who feels well prepared 
to implement NGSS, to identify areas of content weakness 
and professional development needs, and to use the infor- 
mation we have learned from this survey’s results to 
provide recommendations for teacher training as lead 
states begin the NGSS adoption process. 


Literature Review 

Not all teachers are receptive to educational innovations 
(Abrami, Poulsen, & Chambers, 2004). Some embrace the 
innovation with great enthusiasm and incorporate it into 
their teaching while others discard it and return to tradi- 
tional teaching practices after only a few attempts (Lam, 
Cheng, & Choy, 2010). In order for teachers to persist in 
their efforts to adopt new strategies, they need to have the 
expectation that they will succeed (Abrami et al., 2004). 
Academic Standards 

To understand the relevance, potential implications, and 
likely trajectory of the NGSS implementation, it is first 
important to view academic standards within the more 
general historical context of standards-based educational 
reform. This history of academic standards as they are 
currently understood has its roots in a push for educational 
reform that began in the early 1980s, continued in strength 
through the 1990s, and continues to this day (Zuzovsky & 
Libman, 2006). This surge in the popularity of standards- 
based educational reform is often thought of as originating 
with the 1983 publication of A Nation at Risk (The 
National Commission on Excellence in Education, 1983) 
and the resulting nationwide panic over the state of 
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American education and the high school graduate’s ability 
to function as a productive member in a national and 
international society (Vogler & Virtue, 2007). Problems of 
schooling were considered to be due to low accountability 
within the educational system. Standards made educa- 
tional goals more transparent and directly indicate what is 
expected of stakeholders, namely teachers and students 
(Miskel & Ogawa, 1988). By 1989, curriculum standards 
were developed in math, and other subject areas soon 
followed suit (Delandshare & Petrosky, 2004). 

Evolution of standards (state and national) in the STEM 
disciplines. NGSS is a set of science standards. Science 
and its fellow technology, engineering, and mathematics 
(STEM) disciplines are, and have been for a least a decade, 
the focus of tremendous concern and reform effort from 
educators nationwide (Dugger, 2010; Sanders, 2009). The 
reasons for this are varied. The concern that the United 
States is falling behind other nations of the world in STEM 
areas, and that this lack could potentially manifest as eco- 
nomic disaster in the future (Sanders, 2009) has resulted in 
an influx of federal funding for STEM education and 
research, and the encouragement of organizations (e.g., 
National Science Foundation, or NSF) to pursue this 
research (Dugger, 2010). 

Over the past two decades, state science standards have 
been based on the 1996 National Science Education Stan- 
dards (published by the National Research Council) and 
the 1993 Benchmarks for Scientific Literacy (Labov, 2006). 
Although these national documents do not represent stan- 
dards adopted by all states, they aid in the development of 
state frameworks though this translation and adoption 
process has often proven difficult (Tanner & Allen, 2002). 
Subsequently, the No Child Left Behind Act (NCLB, 
2002) was signed into law. According to the U.S. Depart- 
ment of Education (2002), this Act was meant to change 
the culture of American education by mandating achieve- 
ment and results. These results are monitored via manda- 
tory testing for all students enrolled in grades three 
through eight. NCLB requires that students be tested in 
reading language arts, mathematics, and science. 

Although science is a subject for which NCLB mandates 
annual testing, the testing is not nationalized. NCLB gives 
states the responsibility to set standards, support classroom 
instruction, ensure qualified teachers, and create assess- 
ment tools to measure progress (Marx & Harris, 2006). As 
such, state-designed and implemented standards are gener- 
ally related in content and form, though the degree of this 
interrelation is variable. It is possible that the need to create 
standards that can be measured by high-stakes testing, 
drives states toward producing standards that focus on facts 
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rather than on difficult-to-measure skills such as deep 
thinking and conceptual understanding of the underlying 
processes of science and STEM subjects (Marx & Harris, 
2006). Some claim that high-stakes testing at the secondary 
level prompts teachers to cover massive amounts of infor- 
mation, and so moving beyond an educational experience 
based in anything more that rote memorization is difficult 
(Vogler & Virtue, 2007). 

NCLB and high-stakes testing have been criticized by 


_ both educators and educational researchers. One persistent 


criticism is that the high-stakes nature of these federally 
mandated tests promotes teaching to the test. (Cawelti, 
2006; Flinders, 2005; Guilfoyle, 2006). Science standards 
are to some degree political documents, co-created by 
committees of individuals representing a broad spectrum 
of educational and scientific interest groups. A standards 
document produced in this way represents a group’s best 
effort to find consensus about the essential concepts to be 
learned within scientific subdisciplines. However, these 
standards cannot ensure that the concepts will receive 
similar treatment in all classrooms. 

The developmental trajectory of NGSS. In the summer 
of 2011, a writing team of 41 individuals began work on 
the first draft of the NGSS. As the effort was intended to be 
a collaborative effort of various stakeholders, the docu- 
ment was released for public review multiple times, and 
the writers were charged with being responsive to the 
feedback these drafts received. On April 9, 2013, the final- 
ized NGSS document was released. 

NGSS describe what all students should know and be 
able to do by the time they graduate from high school. 
NGSS are based on learning progressions of core ideas in 
the discipline, concepts that cut across disciplines and 
practices that will allow students to use their disciplinary 
knowledge in meaningful ways. An important advance 
from earlier standards from the National Research Council 
(NRC, 1996), NGSS Science and Engineering Practices 
are clearly identified not as separate learning goals that 
define what students should know about the process of 
science, but rather as ways of identifying the reasoning 
behind, discourse about, and application of the core ideas 
in science (Reiser, Berland, & Kenyon, 2012). 

The science and engineering practices outlined in the 
NGSS are: asking questions and defining problems, devel- 
oping and using models, planning and carrying out inves- 
tigations, analyzing and interpreting data, using 
mathematics and computational thinking, constructing 
explanations and designing solutions, engaging in argu- 
ment from evidence, and obtaining, evaluating, and com- 
municating information (Achieve, Inc., 2013). 
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The process of creating the NGSS was driven by 26 lead 
states. These states contributed resources and support to 
the development process, and are expected to be trailblaz- 
ers in the adoption and implementation of the NGSS. The 
choice to implement curricula and the form these curricula 
will take is consequently at the discretion of individual 
states. 

NGSS and Modeling Instruction 

One cornerstone of NGSS (Achieve, Inc., 2013) is the 
use of eight core science and engineering practices, which 
are identified above. These practices are at the foundation 
of Modeling Instruction, a research-based pedagogy for 
science education, developed in the 1980s. 

Modeling Instruction integrates a student-centered 
teaching method with a model-centered curriculum 
(Hestenes, 1996; Jackson, Dukerich, & Hestenes, 2008). It 
applies structured inquiry techniques to teaching basic 
skills in mathematical modeling, proportional reasoning, 
quantitative estimation, and data analysis, which contrib- 
ute to the development of critical thinking, including the 
ability to formulate hypotheses and evaluate them with 
rational argument and evidence. 

Modeling pedagogy has three components: the models, 
the Modeling Cycle, and classroom discourse manage- 
ment (Hestenes, 1996). An understanding of these compo- 
nents is the pedagogical content knowledge (Shulman, 
1986) needed for successful classroom implementation. A 
model is a representation of structure—a conceptual rep- 
resentation of a real thing (Hestenes, 1987). According to 
Johnson-Laird (1996), mental models have a structure that 
corresponds to the structure of what they represent. He 
defines perception as the transformation of sensory infor- 
mation into a mental model, and defines thinking as the 
manipulation of mental models. The models around which 
learning is centered in modeling are fundamental relation- 
ships among quantities that form the content core of a 
discipline (e.g., physics), and these models are developed 
by students into tools for making sense of physical 
reality—for making predictions and answering questions. 

Modeling is an activity. Based on Karplus’ Learning 
Cycle (Karplus, 1977), the Modeling Cycle (Hestenes, 
1987) is a three-phase process: (a) model construction, 
which takes place in the context of a paradigm lab that 
uncovers a relationship between two physical quantities at 
the beginning of each instructional unit, (b) model valida- 
tion, in which the student refines the basic model that he or 
she has constructed by testing it in different initial condi- 
tions, and (c) model deployment, in which the student uses 
the model to solve problems from a variety of contexts. 
This is analogous to the model-of-model for progres- 
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sion (Streefland, 1985; Treffers, 1987; Van den 
Heuvel-Panhuzen, 2003) described by Dutch Realistic 
Mathematics Education researchers who characterize 
models as bridging objects from informal to formal 
reasoning. 

Others have contributed to current understanding of 
how students develop and use models in K-12 classrooms. 
Modeling research has focused on argumentation in 
science education (Bottcher & Meisert, 2011; Passmore & 
Svoboda, 2012), the role of models and analogies in 
science education (Coll, France, & Taylor, 2005), software 
scaffolds supporting modeling practices (Fretz etal., 
2002), model-based inquiry (Passmore, Stewart, & 
Cartier, 2009), integrating conscious and intuitive knowl- 
edge (Cheng & Brown, 2010), and constructing and revis- 
ing models in the science classroom (Krajcik & Merritt, 
2012). Numerous researchers investigating modeling 
nationally and internationally have influenced the concep- 
tualization of modeling articulated in NGSS (Schwarz 
et al., 2009; Windschitl, Thompson, & Braaten, 2008). 

Teachers learn Modeling Instruction by participating in 
a Modeling Workshop—an intensive, three-week 90-hour 
immersion experience. Teachers participate in laboratory 
investigations and activities, designing experiments, col- 
lecting, analyzing, interpreting data, and engaging in 
classroom discourse to achieve collective sense-making. It 
is only by active participation in the discourse that char- 
acterizes the Modeling learning that a teacher can become 
an effective manager of modeling discourse in his or her 
own classroom. 

The Modeling Instruction Program has been in exis- 
tence since 1990. From its initial beginnings in college and 
high school physics, it has spread across the science dis- 
ciplines into chemistry, biology, physical science, and 
middle school (integrated) science. Quantitative evidence 
supporting the impact of Modeling Workshops has been 
established through study undertaken as part of the Mod- 
eling Workshop Project evaluation and the U.S. Depart- 
ment of Education has identified Modeling Instruction as 
one of two Exemplary K-12 Science Programs. 


Study Overview 

This study examines in-service teacher motivation to 
adopt NGSS science and engineering (S&E) practices and 
their feelings of readiness to implement these practices in 
their classrooms. We attempt to discern the characteristics 
of teachers who feel well prepared to implement NGSS 
practices and could potentially become mentors for their 
peers in the NGSS adoption process. In particular, we 
explore the link between those who identify themselves as 
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“Modelers” (i.e., practitioners of the Modeling Method of 
Instruction and NGSS receptiveness and readiness). 

Survey data for a pilot study were collected from a 
group of 45 middle and high school science teachers from 
four suburban school districts who were participating in a 
Math Science Partnership PD program to develop skill in 
using NGSS science and engineering practices. Data from 
their responses revealed that these science teachers were 
motivated to use NGSS science and engineering practices, 
particularly those trained in the use of Modeling Instruc- 
tion. The pilot study instrument served as the foundation 
for the larger national online survey developed for admin- 
istration in March and April of 2013. 


Method 
Research Design 

This study employed a mixed-method approach and 
thus, the investigator collected, analyzed, and drew infer- 
ences from both quantitative and qualitative data in a 
single study. It involved the assumption that the use of 
quantitative and qualitative approaches in combination 
provides greater understanding of the research problem 
than either approach alone (Creswell & Plano Clark, 2011; 
Johnson, Onwuegbuzie, & Turner, 2007). The survey had 
several open-ended questions and comment boxes to 
enable teachers to provide rich texture around the survey 
responses. Teachers could further discuss NGSS concerns 
(e.g., inadequate training) or identify additional support 
and resources needed for implementation. 

The research design included both quantitative and 
qualitative methods and employed analysis of variance 
(ANOVA) to determine differences between groups and 
qualitative analysis to code, categorize, and andlyze 
teacher comments. The assumptions of homogeneity of 
variance, normality, and independence were tested and 
met. The survey used a 5-point Likert scale to determine 
the extent to which teachers felt they needed professional 
development to prepare for NGSS adoption and to elicit 
current levels of motivation and perceived readiness to use 
NGSS science and engineering practices. A test for inter- 
nal consistency was conducted showing a Cronbach’s 
alpha of .85. This is commonly used as an estimate of the 
reliability of a psychometric test. An open-ended response 
section was provided for teachers to volunteer comments 
with respect to survey items. Research questions that 
guided the study included: 

1. How motivated are in-service 7-12 teachers to use 
NGSS S&E practices? 

a. What are the differences between middle school 
and high school teachers? 
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2. How well prepared do 7-12 teachers feel to use 
NGSS S&E practices? 

a. What are the differences between middle and high 
school teachers? 

3. Do teachers trained in the use of Modeling Instruc- 
tion feel more motivated and prepared to use NGSS S&E 
practices? 

The survey collected information on school size and 
type (public, private, or charter), teacher interest in pro- 
fessional development and the type of PD that appealed to 
them (e.g., STEM and NGSS training), and socio- 
demographic characteristics like age, years of teaching, 
subjects taught, and grade levels taught. Based on pilot 
study findings, it was hypothesized that years of teaching, 
types of courses taught, and a prior training in and teach- 
ing Modeling Instruction would predict perceived readi- 
ness for NGSS. 

Participants and Context 

In-service teachers from across the nation were invited 
to complete the survey. These individuals were employed 
in public, private, and charter schools and represent a 
national pool of teachers who teach primarily science in 
middle school and high school: physics or physical 
science, chemistry, and biology. Teachers who subscribed 
to science listservs were sent a link to the NGSS Readiness 
Survey in March of 2013. The survey received 710 teacher 
responses from across the United States representing 38 
states. 

Quantitative Data 

In order to answer the first research question (How moti- 
vated are in-service 7—12 teachers to use NGSS science and 
engineering practices?), a bank of survey questions was 


provided that included the eight NGSS S&E practices. To 


answer the second question (How prepared do 7—12 teach- 
ers feel to use NGSS S&E practices?), teachers rated their 
perceived success in applying the eight practices in the 
classroom. To answer the third research question (Do teach- 
ers trained in the use of Modeling Instruction feel more 
motivated and prepared to use NGSS S&E practices?), we 
used statistical analyses to look for group differences. 

Data analysis. A 2 x2 factorial ANOVA was used to 
determine specific NGSS practice areas in which Model- 
ers felt more NGSS ready. When there was a statistically 
reliable overall difference in NGSS readiness scores (for 
Modeling or grade level), we used a two-tailed, indepen- 
dent samples /-test to identify the direction of this differ- 
ence. All tests had an alpha level of .05. 
Qualitative Data 

Teachers were given the opportunity to comment on 
their NGSS readiness and motivation in a comment box. 
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Teachers could express any concerns in this area, and as.a 
result, areas of content weakness emerged without 
prompting. The researcher used the constant comparative 
method (Glaser & Strauss, 1967) as a conceptualizing 
method on the first level of abstraction. Teacher comments 
(open-ended comments) were conceptualized line by line. 
The initial phase involved conceptualizing all the incidents 
in the data. The researcher compared data and continually 
modified and sharpened the growing theory at the same 
time. Notes were compared to find differences and consis- 
tencies between codes (e.g., similarity in meanings), 
which helped reveal categories. Data were analyzed using 
a three-step process: data reduction, data display, and con- 
clusion drawing and verification (Miles & Huberman, 
1994). Data reduction helped sort, focus, and condense 
excerpts, which helped organize the data to develop con- 
clusions. Data display enabled review of the reduced data 
so that conclusions could be drawn. Teachers’ excerpts 
formed the basis for identifying categories, themes, and 
assertions. 


Quantitative Results 

Overall, demographic data for the 710 teachers who 
responded to the NGSS national readiness assessment 
revealed that about one third held a bachelor’s degree 
(27%), 68% had a master’s degree, and 5% had a doctorate 
degree. Demographic information is shown in Table 1. 

The majority of respondents taught physics (32%), fol- 
lowed by “science” (23.5%) (presumably an integrated or 
general science course), chemistry (20.4%), biology 
(18.8%), mathematics (2.7%), earth science (2.4%), and 
physical science (0.2%). 
NGSS Teacher Motivation: Middle School and High 
School Differences 

To answer the first research question, “How motivated 
are in-service 7-12 teachers to use NGSS science and 
engineering practices”? (and the sub-question, “What are 
the differences between middle school and high school 
teachers?”’), the study employed a 5-point Likert scale. The 
scale used a range of 5 = “Highly motivated” to 1 = “Not 
motivated.” High school teachers were reported a higher 
degree of motivation to use all eight NGSS science and 
engineering practices than middle school teachers. Results 
are shown in Table 2. 
NGSS Readiness to Use Science and Engineering 
Practices: Teacher Differences 

To answer the second research question, the survey 
included a 5-point Likert scale to determine NGSS readi- 
ness by asking teachers to rate their expectation of success 
in implementing NGSS science and engineering practices 
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Table 1 
Demographic Data for Survey Respondents 
Sex 
Male 43% 
Female 57% 
Degree 
Bachelor’s 27% 
Master’s 68% 
Doctorate 5% 
Certification 
Baccalaureate 49% 
Post-baccalaureate 47% 
None 4% 
School Type 
Public 86% 
Private 10% 
Charter 4% 
School Size 
Small 6% 
Medium 20% 
Large 26% 
Very Large 48% 
Teaching Assignment 
Physics 32% 
Chemistry 20.4% 
Biology 18.8% 
Other Science 2.6% 
Math 127% 
Science (Middle School) 23.5% 
Trained in Modeling Instruction 
Yes 51% 
No 49% 


Note. Total sample size (710 teachers): 51% Modelers (362 
teachers), 49% non-Modelers (348 teachers). 


(5 = “Highly certain of success” to 1 = “Cannot expect 
success”). High school teachers’ responses indicated they 
felt more prepared to implement NGSS science and engi- 
neering practices than middle school teachers (Table 3). 
NGSS Readiness: an Examination of Modelers and 
Non-Modelers 

Of the respondents, 51% (362 teachers) indicated they 
used the Modeling Method of Instruction (indicated “Yes” 
on the survey), while 49% were non-Modelers (indicated 
“No”). The middle school Modelers had been teaching the 
same number of years (M = 16, SD = .88), as high school 
Modelers (M = 15.82, SD =.89) and practiced Modeling 
Instruction in the classroom for the same number of years 
(M=5.8, SD =.58) as high school Modelers (M = 6.04, SD 
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Table 2 


NGSS Eight Practices and Motivation: Middle School and High School 
Teacher Differences 





Middle School High School 


Mean (SD) Mean (SD) 

Motivation 

Asking questions and defining 4.28(.717) 4.54* (.647) 
problems 

Developing and using models 432747)” 4.57* 6617) 

Planning and carrying out 4.24(.870)  4.44* (.727) 
investigations 

Analyzing and interpreting data 4.31(.818) 4.62* (.600) 

Using math and computational 4.17 (.854)  4.53* (.686) 
thinking 

Constructing explanations and 3:901G831)5 3-74.29*F (833) 
designing 

Solutions 

Engaging in argument from 4.23 (.807)  4.50* (.733) 
evidence 

Obtaining, evaluating, 4.39 (.720)  4.54* (.614) 


communicating information 
* P < .05. Total sample size (710 teachers). 


Table 3 
NGSS Practices and Readiness: Middle and High School Teacher Differences 


Middle School High School 


Mean (SD) Mean (SD) 

Teachers would have success in: 

Asking questions and 4.09 (.763) 4.38* (.684) 
defining problems 

Developing and using 4.05 (.869) 4.30* (.732) 
models 

Planning and carrying 3.99 (.867) 4.21* (.791) 
out investigations 

Analyzing and 4.10 (.831) 4.45* (.658) 
interpreting data 

Using mathematics and 3.90 (.859) 4.31* (.771) 
computational 
thinking 

Constructing 231,936) 3.98* (.901) 
explanations and 
designing solutions 

Engaging in argument 3.99 (.874) 4.34* (.787) 
from evidence 

Obtaining, evaluating, 4.19 (.818) 4.42* (.684) 


and communicating 
information 


Note. Total respondents (710 teachers). Degree of predicted 
success (5 = highly certain to 1 = cannot expect success). 
Pe OT. 


= .68). Middle school Modelers had participated in an 
average of 64 hours of Modeling PD; the average high 
school Modeler had completed 117 hours of Modeling PD 
as seen in Table 4. 

Training in modeling instruction. There are a variety 
of professional development experiences that provide 
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Table 4 
Modeler Demographics 


Middle School High School 
Modelers Mean Modelers Mean 


Hours/Modeling workshops 64 


117 
Years using modeling 5.8 6 
instruction 
Years of teaching 16 15.87 


Note. Modeling respondents (362 teachers). 


teachers with training in the Modeling Method of Instruc- 
tion, from one hour after school sessions, or four to eight- 
hour sessions at professional conferences, to multi-week 
Modeling Workshops. Full Modeling Workshops are typi- 
cally 60-90 hours in length (two to three weeks). This 
sample of Modelers’ median of 90 hours indicates that the 
typical respondent attended a 90-hour workshop. While 
the high school Modelers, on average, attended a three- 
week workshop, the middle school Modelers attended a 
two-week workshop. 
NGSS Motivation: Modelers and Non-Modelers 

To answer the third research question, “Do teachers 
trained in the use of Modeling Instruction feel more moti- 
vated and prepared to use NGSS science and engineering 
practices?” Modelers expressed more motivation to use 
NGSS S&E practices than non-Modelers, and the differ- 
ence was significant for all items (P < .01) as seen in 
Table 5. 
NGSS Performance: Modelers and Non-Modelers 

Those who use Modeling Instruction rated themselves 
as better prepared to utilize NGSS science and engineering 
practices than non-Modelers, and the differences were sig- 


nificant for all eight NGSS practices (P < .05). Teacher 


perceptions of NGSS readiness are shown in Table 6. 

Modelers rated themselves as better prepared to imple- 
ment NGSS science and engineering practices, including 
“Constructing explanations for science and designing 
solutions for engineering” reflected in the means for Mod- 
elers (M = 4.10, SD = .866) and non-Modelers (M = 3.84, 
SD = .954). In addition to probing for readiness to utilize 
the NGSS practices, the survey included one item that 
specifically elicited readiness to incorporate engineering 
problem solving into science instruction. Modelers felt 
more prepared for “Effectively using engineering prob- 
lems to help students understand science concepts” than 
the non-Modelers, and the difference was significant. 
Grade Levels and Modeling: Differences Between 
Groups 

Results from the ANOVA showed that there were no 
statistically reliable interactions between Modeling and 


421 


Table 5 


How Motivated Are Teachers Who Model Versus Those Who Don’t Model? 
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‘ 


Modelers | Non-Modelers 
Mean (SD) Mean (SD) 
I am motivated to: 
Ask questions and define 4.58 (0.616)  4.38* (0.692) 
problems 
Develop and use models 4.69 (0.545)  4.14* (0.744) 
Plan and carry out 4.60 (0.613)  4.29* (0.824) 
investigations 
Analyze and interpret data 4.67 (0.567)  4.35* (0.699) 
Use math and computational 4.51 (0.692)  4.27* (0.814) 
thinking 
Construct explanations, design 4.38 (0.780) 4.09* (0.893) 
solutions 
Engage in argument from 4.57 (0.672) 4.32* (0.797) 
evidence 
Obtain, evaluate, and 4.59 (0.585)  4.41* (0.682) 


communicate information 


Note. Total respondents (710 teachers): 51% Modelers (362 
teachers), 49% non-Modelers (348 teachers). 
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Table 6 


Success in NGSS Practices: Modelers Versus Non-Modelers 


Modelers Non-Modelers 
Mean (SD) Mean (SD) 
Teachers would have success in: 
Asking questions and defining 4.38 (0.691) 4.26* (0.752) 
problems 
Developing and using models 4.37 (0.723) 4.11* (0.794) 
Planning and carrying out 4.24 (0.794) 4.09* (0.823) 
investigations 
Analyzing and interpreting data 4.47 (0.653) 4.27* (0.756) 
Using math and computational 4.38 (0.722) 3.90* (0.039) 
thinking 
Constructing explanations for 4.10 (0.866) 3.84* (0.954) 
science and designing 
solutions for engineering 
Engaging in argument from 4.38 (0.791) 4.11* (0.837) 
evidence 
Obtaining, evaluating, and 4.44 (0.682) 4.30* (0.752) 
communicating information 
Effectively using engineering 3.65 (0.969) 3.47* (1.018) 


problems to help understand 
science concepts 


Note. Degree of predicted success (5 = highly certain to 1 = 
cannot expect success). Total sample size (710 teachers): 51% 
Modelers (362 teachers), 49% non-Modelers (348 teachers). 


Pee (5) 


grade level for any of the eight selected NGSS practices. 


That is, the effect of Modeling does not depend on grade 
level. Thus, we would expect to find reliably higher means 
for Modelers in both middle and high school teachers. The 
researchers classified teachers who indicated that they 
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taught in grades seven and eight as teaching at the “middle 
school” level and those teaching in grades nine, ten, 
eleven, and twelve as teaching at the “high school” level. 

Modelers in middle school and in high school also rated 
themselves as more “prepared” to implement the NGSS 
science and engineering practices in five of the eight prac- 
tices: Developing and using models (NGSS S&E #2), ana- 
lyzing and interpreting data (NGSS S&E #4), using 
mathematics.and computational thinking (NGSS S&E 5), 
constructing explanations and designing solutions (NGSS 
S&E #6), and engaging in argument from evidence (NGSS 
S&E #7). Modelers from middle school and high school 
had similar results in developing and using models and 
using math and computational thinking. Grade level dif- 
ferences are shown in Table 7. 

In order, by NGSS practices: High school Modelers 
rated their NGSS readiness reliably higher (/ = 4.40, SD 
= .73) than did high school non-Modelers (M = 4.20, SD =. 
72) in developing and using models (P < .001). High 
school Modelers felt better prepared (M = 4.51, SD =.64) 
than did high school non-Modelers (M = 4.36, SD = .67) in 
analyzing and interpreting data (P = .01). High school 
Modelers also had higher scores (M = 4.42, SD =.71) than 
did non-Modelers (M = 4.17, SD = .83) in using math- 
ematics and computational thinking (P < .001). Finally, 
High school Modelers also rated themselves as more pre- 
pared in constructing explanations and deoneienay solutions 
(P = .01) than non-Modelers. 


Qualitative Results 

Elements Needed to Facilitate NGSS Adoption 

Qualitative comments indicate that although most teach- 
ers are positive about NGSS, they are anxious about inad- 
equate training, limited instructional time, and lack of 
resources, all barriers to implementation. Teachers’ com- 
ments in this study echoed impediments previously iden- 
tified in a study of elementary teachers, such as 
insufficient preparation, limited instructional time devoted 
to science, and the lack of physical resources for effective 
science education (Trygstad et al., 2013). In addition to 
fears about the implementation, the comments 
(commenters’ names have been replaced by pseudonyms 
in the sections below) also revealed that many teachers 
considered the standards to be too complex in some case 
exceeding the knowledge and training of the educators 
who would be expected to teach the content mandated. 

Professional development needed. Although at first 
glance it appears that most teachers rate themselves as 
reasonably competent to incorporate NGSS S&E practices 
into their teaching, the accompanying comments by 
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Table 7 
Middle and High School: Modelers and Non-Modeler Differences 


Group 


Asking questions and defining problems 

Developing and using models 

Planning and carrying out experiments 

Analyzing and interpreting data 

Using mathematics and computational thinking 
Constructing explanations and designing solutions 
Engaging in argument from evidence 

Obtaining, evaluating, and communicating information 


High School 


High School 


Middle School 


Middle School 


Non-Modeler Modeler Non-Modeler Modeler Mean 

Mean (SD) Mean (SD) Mean (SD) (SD) 
4.34 (.68) 4.41 (.69) 4.0 (.79) 4.1 (.70) 
4.20 (.72) 4.40 (.73)*** 3.40 (.93) 4.27 (.67)* 
4.14 (.77) 4.26 (.80) 30) 4.92) 4.10 (.74) 
4.36 (.67) 4.51 (.64)** 4.10 (.88) 4.20 (.68) 
4.17 (.83) 4.42 (.71)*** 3.84 (.90) 4.15 (.73)* 
3.92 (.93) 4.20 (.88)** 3.67 (.99) 3.95 (.74) 
4.31 (.77) 4.37 (.80) 3.92 (.91) 4.23 (.73)* 
4.36 (.68) 4.46 (.69) 4.12 (.88) 4.37 (.62) 


Note. Degree of predicted success (5 = highly certain to 1 = cannot expect success). Total sample size (710 teachers): 51% Modelers 


(362 teachers), 49% non-Modelers (348 teachers). 
Ee ee tS re OU L. 


respondents indicate there is concern and in some cases 
outright anxiety about expectations with respect to their 
use of and success with S&E practices. Teachers see a 
need for PD at both middle school and high school levels 
to facilitate successful NGSS adoption. The majority 
(84.9%) of teachers in this national sample are interested 
in pursuing future NGSS PD, but most indicated they 
would do this only if their school or district was willing to 
pay for it (70%). Teachers indentified the content focus for 
PD needed to improve NGSS readiness as follows: engi- 
neering (65.6%), technology (50%), science (45.5%), and 
mathematics (26.2%). Engineering emerged as the content 
area of greatest need, which is reflected in this excerpt, “I 
have only a loose grasp on the engineering and design part 
of the standards and am in desperate need of a concise 
overview of what they are and how to incorporate them 
into my teaching” (C. Valdez, survey response, April 13, 
2013). Another teacher stated, “The engineering practices 
in the NGSS standards intrigue me, but I’m a science 
teacher, not an engineer. I do build projects in my classes 
to relate the physics to the practical aspects of engineering, 
but I have no training in this” (S. Rymes, survey response, 
April 15, 2913). As noted by T. Myers (survey response, 
April 28, 2013), “As a whole, most teachers will be less 
comfortable with the science and engineering practices 
than the disciplinary core ideas. There will be teacher 
misconceptions on what the performance expectations 
really are. Without profession development and supports, 
teachers will not be effective.” A final comment on the 
critical need for training showed a middle school teacher’s 
concern, “I think NGSS is a good idea but if middle- 
school teachers are not trained, it will fail. Current stan- 
dardized testing pulls the curriculum in different path. I 
see these two as opposing forces” (A. Boss, survey 
response, April 27, 2013). 
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More instructional time and resources 
needed. Open-ended survey feedback revealed other 
concerns about NGSS implementation: limited instruc- 
tional time and resources. A large response rate from 
middle school teachers (68%) indicated that teachers per- 
ceived time as a barrier to successful NGSS implemen- 
tation. Teachers recognized that cross-cutting concepts 
were important but realized that reform is needed to allo- 
cate more instructional time. One participant (J. Mendez, 
survey response, April 16, 2013) echoed this sentiment, 
“More time is needed to teach a course using the inquiry 
method and NGSS will require more time and 
resources.” Another teacher talked about adding engi- 
neering to his instruction, which will also require more 
time and materials, “I hope adding engineering doesn’t 
mean we must always build something. Engineering 
practices can be met by applying knowledge and skills to 


real world scenarios. Building and design will cost a lot 


of money and take up a lot of my current class time” 
(J. Johnson, survey response, April 10, 2013). The lack 
of resources and time represent a significant impediment 
to the NGSS as noted in prior research; in addition 
to the new standards, teachers will have to contend 
with not having the necessary facilities, equipment, 
materials, or time for science teaching (Trygstad et al., 
2013). : 

A theme running throughout the comments reinforced 
the notion that teachers lacked an engineering background, 
and many suggested they needed additional training, “I 
like that the focus has been shifted from ‘learning’ science 
to ‘doing’ science. Yet, most of us that will be implement- 
ing these standards have no engineering background or 
training and are ill prepared to effectively implement engi- 
neering practices” (J. Beck, survey response, April 4, 
2013). 
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Modeling instruction well aligned — with 
NGSS. Teacher comments linked NGSS with Modeling 
Instruction, as one participant indicated, “Modeling is well 
aligned with the skills and practices outlined by NGSS” (J. 
Smith, survey response, April 29, 2013). Another teacher 
echoed this alignment, reflected in this excerpt, “I’m 
excited about the NGSS. I am a teacher using the modeling 
method so I am comfortable with models and scientific 
thinking in my classrooms” (A. Banks, survey response, 
April 17, 2013). Comments linking NGSS to Modeling 
Instruction were frequent in the subset of Modeling 
respondents, as one noted, “The linkage between NGSS 
and Modeling is obvious for those who practice Modeling. 


Discussion 

This study sought to determine how prepared and moti- 
vated middle and high school teachers are to adopt and use 
NGSS S&E practices in their classrooms. Research sug- 
gests that there are likely to be significant challenges to 
overcome for states, districts, and schools that adopt 
NGSS (Trygstad et al., 2013). Three barriers that have 
already been identified by prior research are: the need for 
time, resources, and training. In addition to concerns about 
limited school resources and instructional time (e.g., the 
difference between how science is currently taught and 
what NGSS will mandate), the third area of concern is 
teacher preparation (Cooper, 2013; Trygstad et al., 2013), 
which has been examined at both the middle and high 
school levels in this study. We found that high school 
teachers who have completed an average of 90 hours of 
professional development in Modeling Instruction (the 
typical three-week workshop) feel significantly more 
motivated and better prepared for NGSS than high school 
teachers who are non-Modelers. Middle school teachers, 
with an average of.64 hours of professional development 
in Modeling Instruction, felt better prepared in a number 
of NGSS practice areas than non-Modeler middle school 
teachers. Both high school and middle school teachers 
identified a need for preparation in engineering content. 
Perceived Success in Utilizing NGSS Science and 
Engineering Practices 

Current science teaching practices often emphasize the 
memorization of facts, yet NGSS emphasizes the primacy 
of the active construction by students of conceptual knowl- 
edge by “doing science” via science and engineering prac- 
tices. Modeling Instruction shifts the paradigm of how 
science is taught to a pedagogy that is well aligned with 
NGSS S&E practices. Teachers trained in Modeling 
Instruction feel better prepared and more likely to succeed 
in implementing these practices in their classrooms. Mod- 
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elers say that they are already routinely incorporating S&E 
practices into their daily classroom activities. 
Limitations 

This survey was distributed via a collection of local and 
national listservs to which middle and high science teach- 
ers subscribe. Respondents were self-selected as there was 
no incentive for survey completion. Therefore, these 
survey results are generalizable to the population of teach- 
ers who would routinely read such a list—teachers who 
care enough about their teaching and learning to look to 
their peers for ways to improve their classroom practice. 


Implications 

Modelers express more motivation and readiness to use 
NGSS S&E practices. What might this mean to the schools 
and districts they serve? This teacher community offers a 
profile of well-prepared, confident teachers who could 
become leaders in NGSS teacher professional develop- 
ment. In addition, they could be enlisted as peer mentors in 
schools and districts to help with the transition to the new 
standards. 

As NGSS move toward large-scale adoption and imple- 
mentation, it is critical that educational leaders nationwide 
understand what these standards and the changes they 
portend will mean for the teachers who must work with 
them. To this end, our study attempts to identify factors 
affecting implementation and teacher readiness, particu- 
larly in the area of science and engineering practices. Our 
results support and build upon prior research on elemen- 
tary teachers, adding to the extant literature on NGSS 
readiness. Similar to the findings of Trygstad et al. (2013), 
our data indicate a need for focused professional develop- 
ment for both high school and middle school teachers. 
Among other things, we have found that teachers are con- 
cerned about receiving additional training and education 
in engineering. Teachers also perceived time as a barrier to 
successful NGSS implementation. Teachers recognized 
that cross-cutting concepts were important but realized 
that reform is needed to allocate more instructional 
time. 

We have found significant links between teachers’ use of 
Modeling Instruction and their confidence in using NGSS 
science and engineering practices. Teachers with training 
in Modeling Instruction feel more motivated and better 
prepared to implement NGSS S&E practices in their class- 
rooms. In fact, they are already doing so. Seventy Model- 
ing Workshops held nationwide each summer serve 1,000 
physics, chemistry, biology, and physical science teachers 
at 20-25 host institutions (universities and school dis- 
tricts). This professional development network is a 


Volume 115 (8) 


National Study on NGSS Readiness 


resource for middle school and high school teacher pro- 
fessional development in NGSS S&E practices that is 
already in place nationwide. 

The Modeling teacher community numbers over 7,000 
at present. Two hundred Modeling teachers have received 
training to lead Modeling Workshops. Thus, the excess 
capacity already exists in the Modeling teacher commu- 
nity to double the number of Modeling Workshops that 
serve secondary science teachers each summer. 

The findings of this study indicate that in-service teach- 
ers believe they need training to adequately implement 
NGSS S&E practices. Those responsible for ensuring suc- 
cessful implementation need to heed this call by teachers. 
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PROBLEMS 


sec Rye) Ted Eisenberg, Section Editor 


This section of the Journal offers readers an opportunity to exchange interesting mathematical problems and solutions. 
Please send them to Ted Eisenberg, Department of Mathematics, Ben-Gurion University, Beer-Sheva, Israel or fax to 
972-86-477-648. Questions concerning proposals and/or solutions can be sent via e-mail to eisenbt@013.net. Solutions 
to previously stated problems can be seen at http://www.ssma.org/publications. 





Solutions to the problems stated in this issue should be posted before February 15, 2016 


¢ 5373: Proposed by Kenneth Korbin, New York, NY 


Given the equation ee ee ean =Vxt+ yv5 : 
V343-147V5 —V315—135V5 


Find positive integers x and y. 


¢ 5374: Proposed by Roger Izard, Dallas, TX 
In a certain triangle, three circles are tangent to the incircle, and all of these circles are tangent to two sides of the 
triangle. Derive a formula which gives the radius of the incircle in terms of the radii of these three circles. 


¢ 5375: Proposed by Michael Brozinsky, Central Islip, NY 
Determine whether or not there exist nonzero constants a and b such that the conic whose polar equation is 


* \ sin(20) — bcos(20) 


¢ 5376: Proposed by Arkady Alt, San Jose, CA 
Let Gy, Gy, - =|, Gy, D>; ~.> ,.D, De positive teal numbers such that.a, < ay <). 2 <@, and a, €.(b;, b,,),7 = 1,2, 273 
nm vet . 


has a rational eccentricity. 


oe (x=) (x= bo) +(x bn) 
(x= a; )(4— ay) .(x —a,) 


Prove that F(x) < 0 for any x € Dom(F). 


¢ 5377: Proposed by José Luis Diaz-Barrero, Barcelona Tech, Barcelona, Spain 
Show that if A, B, C are the measures of the angles of any triangle ABC and a, b, c the measures of the length of its sides, 


then holds 


a +b 





sin(A—B). 


[[ sin’? (4-B)s ¥ 


cyclic cyclic 


¢ 5378: Proposed by Ovidiu Furdui, Technical University of Cluj-Napoca, Cluj-Napoca, Romania 
Let k= 1 be an integer. Calculate 
[ean ne dx. 
0 e~—l 
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